[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 445137141 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002369] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007464] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013066] *** VALIDATE selinux *** [ 0.021000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025045] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028116] *** VALIDATE tmpfs *** [ 0.029465] *** VALIDATE proc *** [ 0.030213] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.032268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035030] Spectre V2 : User space: Vulnerable [ 0.036011] Speculative Store Bypass: Vulnerable [ 0.039430] debug: unmapping init [mem 0xffffffff92059000-0xffffffff92060fff] [ 0.041187] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042723] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043024] ... version: 2 [ 0.044015] ... bit width: 48 [ 0.045015] ... generic registers: 4 [ 0.046015] ... value mask: 0000ffffffffffff [ 0.047015] ... max period: 00007fffffffffff [ 0.048017] ... fixed-purpose events: 3 [ 0.049016] ... event mask: 000000070000000f [ 0.051241] rcu: Hierarchical SRCU implementation. [ 0.053507] smp: Bringing up secondary CPUs ... [ 0.054618] x86: Booting SMP configuration: [ 0.055030] .... node #0, CPUs: #1 #2 #3 [ 0.059010] smp: Brought up 1 node, 4 CPUs [ 0.061016] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.231880] node 0 deferred pages initialised in 167ms [ 0.234221] devtmpfs: initialized [ 0.236328] x86/mm: Memory block size: 128MB [ 0.241461] gcov: version magic: 0x41383552 [ 0.244376] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.245079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246265] pinctrl core: initialized pinctrl subsystem [ 0.247173] [ 0.247662] ************************************************************* [ 0.248091] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249015] ** ** [ 0.250020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.251013] ** ** [ 0.252013] ** This means that this kernel is built to expose internal ** [ 0.253012] ** IOMMU data structures, which may compromise security on ** [ 0.254013] ** your system. ** [ 0.255014] ** ** [ 0.256015] ** If you see this message and you are not debugging the ** [ 0.257012] ** kernel, report this immediately to your vendor! ** [ 0.258017] ** ** [ 0.259016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.260016] ************************************************************* [ 0.261525] NET: Registered protocol family 16 [ 0.263499] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.266077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.268089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.272173] cpuidle: using governor menu [ 0.273875] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.276601] PCI: Using configuration type 1 for base access [ 0.277190] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.287168] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.290033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.293172] cryptd: max_cpu_qlen set to 1000 [ 0.296176] ACPI: Added _OSI(Module Device) [ 0.297020] ACPI: Added _OSI(Processor Device) [ 0.299019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.301021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.306190] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.311427] ACPI: Interpreter enabled [ 0.313071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.314015] ACPI: Using IOAPIC for interrupt routing [ 0.316153] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.323594] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.335251] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.337053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.340019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.343152] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.347657] acpiphp: Slot [2] registered [ 0.349139] acpiphp: Slot [5] registered [ 0.351116] acpiphp: Slot [6] registered [ 0.352112] acpiphp: Slot [3] registered [ 0.353083] acpiphp: Slot [4] registered [ 0.354113] acpiphp: Slot [7] registered [ 0.356098] acpiphp: Slot [8] registered [ 0.357130] acpiphp: Slot [9] registered [ 0.359087] acpiphp: Slot [10] registered [ 0.360082] acpiphp: Slot [11] registered [ 0.361099] acpiphp: Slot [12] registered [ 0.363104] acpiphp: Slot [13] registered [ 0.364092] acpiphp: Slot [14] registered [ 0.365121] acpiphp: Slot [15] registered [ 0.367107] acpiphp: Slot [16] registered [ 0.368109] acpiphp: Slot [17] registered [ 0.369084] acpiphp: Slot [18] registered [ 0.371087] acpiphp: Slot [19] registered [ 0.372122] acpiphp: Slot [20] registered [ 0.373087] acpiphp: Slot [21] registered [ 0.374123] acpiphp: Slot [22] registered [ 0.376112] acpiphp: Slot [23] registered [ 0.377100] acpiphp: Slot [24] registered [ 0.379094] acpiphp: Slot [25] registered [ 0.380094] acpiphp: Slot [26] registered [ 0.382096] acpiphp: Slot [27] registered [ 0.383202] acpiphp: Slot [28] registered [ 0.385134] acpiphp: Slot [29] registered [ 0.387145] acpiphp: Slot [30] registered [ 0.388161] acpiphp: Slot [31] registered [ 0.390068] PCI host bridge to bus 0000:00 [ 0.392030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.394034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.397034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.400036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.403119] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.406032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.408204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.412040] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.416435] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.425019] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.430300] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.433034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.437028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.440031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.444463] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.446682] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.449043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.451707] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.455745] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.464021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.469014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.474215] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.479020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.485021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.496021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.505812] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.510018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.515020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.525030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.533832] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.536581] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.539290] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.543310] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.546539] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.551104] iommu: Default domain type: Passthrough [ 0.553597] SCSI subsystem initialized [ 0.555225] ACPI: bus type USB registered [ 0.556144] usbcore: registered new interface driver usbfs [ 0.558122] usbcore: registered new interface driver hub [ 0.560087] usbcore: registered new device driver usb [ 0.561215] pps_core: LinuxPPS API ver. 1 registered [ 0.563016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.567084] PTP clock support registered [ 0.569063] EDAC MC: Ver: 3.0.0 [ 0.570662] PCI: Using ACPI for IRQ routing [ 0.572951] NetLabel: Initializing [ 0.575014] NetLabel: domain hash size = 128 [ 0.576010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.578101] NetLabel: unlabeled traffic allowed by default [ 0.581088] vgaarb: loaded [ 0.582440] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.584015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.591000] clocksource: Switched to clocksource kvm-clock [ 0.714565] VFS: Disk quotas dquot_6.6.0 [ 0.716435] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.719315] *** VALIDATE ramfs *** [ 0.720708] *** VALIDATE hugetlbfs *** [ 0.722629] pnp: PnP ACPI init [ 0.725542] pnp: PnP ACPI: found 6 devices [ 0.741724] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.745355] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.747756] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.750353] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.752807] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.755613] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.758624] NET: Registered protocol family 2 [ 0.761299] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.765883] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.770041] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.775856] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.779635] TCP: Hash tables configured (established 65536 bind 65536) [ 0.783039] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.786520] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.789655] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.792970] NET: Registered protocol family 1 [ 0.795900] RPC: Registered named UNIX socket transport module. [ 0.797854] RPC: Registered udp transport module. [ 0.799079] RPC: Registered tcp transport module. [ 0.800311] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.802153] NET: Registered protocol family 44 [ 0.804041] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.806409] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.808673] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.811120] PCI: CLS 0 bytes, default 64 [ 0.812895] Unpacking initramfs... [ 2.227645] debug: unmapping init [mem 0xffff9f83fcc64000-0xffff9f83fffcffff] [ 2.231583] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.234188] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.237417] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.705205] Initialise system trusted keyrings [ 2.707177] Key type blacklist registered [ 2.709263] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.718489] zbud: loaded [ 2.721436] *** VALIDATE nfs *** [ 2.722564] *** VALIDATE nfs4 *** [ 2.723812] pstore: using deflate compression [ 2.727027] Platform Keyring initialized [ 2.831408] NET: Registered protocol family 38 [ 2.833311] Key type asymmetric registered [ 2.834818] Asymmetric key parser 'x509' registered [ 2.836774] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.840100] io scheduler mq-deadline registered [ 2.842067] io scheduler kyber registered [ 2.843818] io scheduler bfq registered [ 2.846037] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.849224] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.852442] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.855605] ACPI: Power Button [PWRF] [ 2.861780] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.868833] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.880281] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.906980] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.941422] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.948389] Non-volatile memory driver v1.3 [ 2.950211] Linux agpgart interface v0.103 [ 2.985217] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.988515] vda: detected capacity change from 0 to 74702848 [ 3.003695] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.007131] vdb: detected capacity change from 0 to 1073741824 [ 3.013995] libphy: Fixed MDIO Bus: probed [ 3.051204] usbcore: registered new interface driver usbserial_generic [ 3.054343] usbserial: USB Serial support registered for generic [ 3.057620] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.062431] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.064517] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.067216] mousedev: PS/2 mouse device common for all mice [ 3.070479] rtc_cmos 00:05: RTC can wake from S4 [ 3.072545] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.074266] rtc_cmos 00:05: registered as rtc0 [ 3.081357] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.086398] intel_pstate: CPU model not supported [ 3.087211] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.093442] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.093467] hid: raw HID events driver (C) Jiri Kosina [ 3.099247] usbcore: registered new interface driver usbhid [ 3.101742] usbhid: USB HID core driver [ 3.103647] drop_monitor: Initializing network drop monitor service [ 3.106076] Initializing XFRM netlink socket [ 3.108369] NET: Registered protocol family 10 [ 3.111419] Segment Routing with IPv6 [ 3.112598] NET: Registered protocol family 17 [ 3.115853] mpls_gso: MPLS GSO support [ 3.120925] RAS: Correctable Errors collector initialized. [ 3.123385] AVX version of gcm_enc/dec engaged. [ 3.125266] AES CTR mode by8 optimization enabled [ 3.198499] sched_clock: Marking stable (3198482973, 0)->(4131305324, -932822351) [ 3.201244] registered taskstats version 1 [ 3.203209] Loading compiled-in X.509 certificates [ 3.205749] zswap: loaded using pool lzo/zbud [ 3.229830] Key type big_key registered [ 3.243905] Key type encrypted registered [ 3.245750] ima: No TPM chip found, activating TPM-bypass! [ 3.248290] ima: Allocated hash algorithm: sha1 [ 3.249992] ima: No architecture policies found [ 3.252084] evm: Initialising EVM extended attributes: [ 3.254179] evm: security.selinux [ 3.255530] evm: security.ima [ 3.256763] evm: security.capability [ 3.258172] evm: HMAC attrs: 0x1 [ 3.260425] rtc_cmos 00:05: setting system clock to 2026-08-15 08:08:03 UTC (1786781283) [ 3.266659] debug: unmapping init [mem 0xffffffff93003000-0xffffffff931fffff] [ 3.269994] debug: unmapping init [mem 0xffffffff91d82000-0xffffffff92058fff] [ 3.279074] Write protecting the kernel read-only data: 28672k [ 3.282466] debug: unmapping init [mem 0xffffffff90403000-0xffffffff905fffff] [ 3.285588] debug: unmapping init [mem 0xffffffff90d14000-0xffffffff90dfffff] [ 3.319886] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.328918] systemd[1]: Detected virtualization kvm. [ 3.331041] systemd[1]: Detected architecture x86-64. [ 3.332654] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.358587] systemd[1]: No hostname configured. [ 3.360364] systemd[1]: Set hostname to . [ 3.362467] random: systemd: uninitialized urandom read (16 bytes read) [ 3.364864] systemd[1]: Initializing machine ID from random generator. [ 3.487497] random: systemd: uninitialized urandom read (16 bytes read) [ 3.490741] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.495682] random: systemd: uninitialized urandom read (16 bytes read) [ 3.502592] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 3.510140] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.047662] device-mapper: uevent: version 1.0.3 [ 4.049641] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. [ 4.266014] random: fast init done Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.670848] virtio_net virtio0 ens2: renamed from eth0 [ 4.734186] scsi host0: ata_piix [ 4.753973] scsi host1: ata_piix [ 4.755868] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.761238] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.071174] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.762407] random: crng init done [ 9.765863] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.602755] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.067119] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.336961] SELinux: Disabled at runtime. [ 12.397581] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.409513] systemd[1]: Detected virtualization kvm. [ 12.412470] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.938860] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.942665] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.947606] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.953410] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.957410] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.967609] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.976547] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ 13.143164] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.602577] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.961419] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.999312] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.158840] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.170083] EDAC sbridge: Ver: 1.1.2 [ 15.367169] Key type dns_resolver registered [ 15.671649] NFS: Registering the id_resolver key type [ 15.673592] Key type id_resolver registered [ 15.675064] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg232-client login: [ 44.175907] libcfs: loading out-of-tree module taints kernel. [ 44.300273] Key type ._llcrypt registered [ 44.302193] Key type .llcrypt registered [ 44.688719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 44.696071] alg: No test for adler32 (adler32-zlib) [ 45.802459] Lustre: Lustre: Build Version: 2.17.57_1_g05abe65 [ 46.175312] LNet: Added LNI 192.168.202.32@tcp [8/256/0/180] [ 47.808167] Key type lgssc registered [ 48.614246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 117.365012] hrtimer: interrupt took 2058573 ns [ 219.318720] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 225.655942] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.495125] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing check_logdir /tmp/testlogs/ [ 243.717648] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing yml_node [ 245.216213] Lustre: lustre-OST0000-osc-ffff9f8461000800: disconnect after 23s idle [ 249.205391] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 252.294256] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 255.426612] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 257.484805] Lustre: DEBUG MARKER: -----============= acceptance-small: recovery-small ============----- Sat Aug 15 04:12:15 EDT 2026 [ 273.874323] Lustre: DEBUG MARKER: excepting tests: 136 [ 276.398607] Lustre: DEBUG MARKER: === recovery-small: start setup 04:12:34 (1786781554) === [ 284.039097] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing check_config_client /mnt/lustre [ 306.862886] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 320.940824] Lustre: DEBUG MARKER: === recovery-small: finish setup 04:13:19 (1786781599) === [ 322.763493] Lustre: DEBUG MARKER: == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 04:13:21 (1786781601) [ 340.448168] Lustre: 10820:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786781604/real 1786781604] req@ffff9f84594f2680 x1873576015707264/t0(0) o700->lustre-MDT0000-mdc-ffff9f8461000800@192.168.202.132@tcp:30/10 lens 264/248 e 0 to 1 dl 1786781620 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 340.476229] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection to lustre-MDT0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 340.510635] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection restored to 192.168.202.132@tcp (at 192.168.202.132@tcp) [ 357.856549] Lustre: 10841:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786781622/real 1786781622] req@ffff9f8460e3d180 x1873576015709568/t0(0) o36->lustre-MDT0000-mdc-ffff9f8461000800@192.168.202.132@tcp:12/10 lens 520/576 e 0 to 1 dl 1786781638 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 357.897192] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection to lustre-MDT0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 357.950137] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection restored to 192.168.202.132@tcp (at 192.168.202.132@tcp) [ 375.776170] Lustre: 10867:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786781640/real 1786781640] req@ffff9f8458f1dc00 x1873576015711360/t0(0) o101->lustre-MDT0000-mdc-ffff9f8461000800@192.168.202.132@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786781656 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:0 [ 375.797458] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection to lustre-MDT0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 375.894698] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection restored to 192.168.202.132@tcp (at 192.168.202.132@tcp) [ 394.208179] Lustre: 10888:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786781658/real 1786781658] req@ffff9f8460e3f480 x1873576015714176/t0(0) o36->lustre-MDT0000-mdc-ffff9f8461000800@192.168.202.132@tcp:12/10 lens 488/512 e 0 to 1 dl 1786781674 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:4294967295 [ 394.221168] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection to lustre-MDT0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 394.246932] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection restored to 192.168.202.132@tcp (at 192.168.202.132@tcp) [ 413.665728] Lustre: 10915:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786781677/real 1786781677] req@ffff9f8460e3f480 x1873576015715968/t0(0) o34->lustre-MDT0000-mdc-ffff9f8461000800@192.168.202.132@tcp:12/10 lens 472/728 e 0 to 1 dl 1786781693 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 413.702177] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection to lustre-MDT0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 413.768304] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection restored to 192.168.202.132@tcp (at 192.168.202.132@tcp) [ 432.608458] Lustre: 10935:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786781696/real 1786781696] req@ffff9f8460e3c380 x1873576015717760/t0(0) o34->lustre-MDT0000-mdc-ffff9f8461000800@192.168.202.132@tcp:12/10 lens 472/728 e 0 to 1 dl 1786781712 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 432.626716] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection to lustre-MDT0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 432.657423] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection restored to 192.168.202.132@tcp (at 192.168.202.132@tcp) [ 440.055532] Lustre: DEBUG MARKER: == recovery-small test 4: open: drop req, drop rep ======= 04:15:19 (1786781719) [ 457.696160] Lustre: 11548:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786781721/real 1786781721] req@ffff9f8460e3ed80 x1873576015720704/t0(0) o101->lustre-MDT0000-mdc-ffff9f8461000800@192.168.202.132@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786781737 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 [ 457.744108] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection to lustre-MDT0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 457.807019] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection restored to 192.168.202.132@tcp (at 192.168.202.132@tcp) [ 485.420766] Lustre: DEBUG MARKER: == recovery-small test 5: rename: drop req, drop rep ===== 04:16:04 (1786781764) [ 502.752341] Lustre: 12178:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786781766/real 1786781766] req@ffff9f8460e3ea00 x1873576015728000/t0(0) o101->lustre-MDT0000-mdc-ffff9f8461000800@192.168.202.132@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786781782 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'mv.0' uid:0 gid:0 projid:0 [ 502.802863] Lustre: 12178:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 502.816472] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection to lustre-MDT0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 502.842792] Lustre: Skipped 1 previous similar message [ 502.880098] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection restored to 192.168.202.132@tcp (at 192.168.202.132@tcp) [ 502.890390] Lustre: Skipped 1 previous similar message [ 529.470300] Lustre: DEBUG MARKER: == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 04:16:48 (1786781808) [ 552.417849] Lustre: lustre-OST0000-osc-ffff9f8461000800: disconnect after 24s idle [ 552.432756] Lustre: Skipped 1 previous similar message [ 585.696249] Lustre: 12866:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786781849/real 1786781849] req@ffff9f8458ebbb80 x1873576015742976/t0(0) o101->lustre-MDT0000-mdc-ffff9f8461000800@192.168.202.132@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786781865 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'unlink.0' uid:0 gid:0 projid:0 [ 585.743446] Lustre: 12866:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 585.751124] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection to lustre-MDT0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 585.773989] Lustre: Skipped 3 previous similar messages [ 585.822198] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection restored to 192.168.202.132@tcp (at 192.168.202.132@tcp) [ 585.833201] Lustre: Skipped 3 previous similar messages [ 611.978690] Lustre: DEBUG MARKER: == recovery-small test 8: touch: drop rep (bug 1423) ===== 04:18:11 (1786781891) [ 636.782416] Lustre: DEBUG MARKER: == recovery-small test 9: pause bulk on OST (bug 1420) === 04:18:35 (1786781915) [ 651.724625] Lustre: DEBUG MARKER: == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 04:18:50 (1786781930) [ 652.330327] Lustre: *** cfs_fail_loc=305, val=0*** [ 667.897534] Lustre: *** cfs_fail_loc=305, val=0*** [ 670.691588] Lustre: lustre-OST0000-osc-ffff9f8461000800: disconnect after 20s idle [ 684.282181] Lustre: *** cfs_fail_loc=305, val=0*** [ 700.685733] Lustre: *** cfs_fail_loc=305, val=0*** [ 716.045534] Lustre: *** cfs_fail_loc=305, val=0*** [ 732.414913] Lustre: *** cfs_fail_loc=305, val=0*** [ 747.767507] Lustre: *** cfs_fail_loc=305, val=0*** [ 764.652947] LustreError: lustre-MDT0000-mdc-ffff9f8461000800: operation ldlm_enqueue to node 192.168.202.132@tcp failed: rc = -107 [ 764.668481] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection to lustre-MDT0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 764.695366] Lustre: Skipped 2 previous similar messages [ 764.725848] LustreError: lustre-MDT0000-mdc-ffff9f8461000800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 764.774316] Lustre: lustre-MDT0000-mdc-ffff9f8461000800: Connection restored to 192.168.202.132@tcp (at 192.168.202.132@tcp) [ 764.790239] Lustre: Skipped 2 previous similar messages [ 774.331330] Lustre: DEBUG MARKER: == recovery-small test 10b: re-send BL AST =============== 04:20:53 (1786782053) [ 800.492086] Lustre: DEBUG MARKER: == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 04:21:18 (1786782078) [ 801.258144] Lustre: *** cfs_fail_loc=305, val=0*** [ 801.260865] Lustre: Skipped 1 previous similar message [ 811.606293] Lustre: DEBUG MARKER: == recovery-small test 10d: test failed blocking ast ===== 04:21:29 (1786782089) [ 815.311910] Lustre: Unmounted lustre-client [ 816.010954] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 817.789203] LustreError: lustre-OST0000-osc-ffff9f84513bc800: operation ost_statfs to node 192.168.202.132@tcp failed: rc = -107 [ 817.798681] LustreError: lustre-OST0000-osc-ffff9f84513bc800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 817.809867] Lustre: 2386:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.132@tcp:/lustre/fid: [0x200000404:0x1:0x0]/ may get corrupted (rc -108) [ 819.636546] Lustre: Unmounted lustre-client [ 827.380472] Lustre: DEBUG MARKER: == recovery-small test 10e: re-send BL AST vs reconnect race 2 ========================================================== 04:21:45 (1786782105) [ 829.078864] Lustre: DEBUG MARKER: SKIP: recovery-small test_10e need two clients [ 830.977679] Lustre: DEBUG MARKER: == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) ========================================================== 04:21:49 (1786782109) [ 859.781359] Lustre: DEBUG MARKER: == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================================================== 04:22:18 (1786782138) [ 860.158383] Lustre: DEBUG MARKER: multiop /mnt/lustre/f12.recovery-small OS_c [ 878.560652] Lustre: 18548:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786782142/real 1786782142] req@ffff9f84594f1c00 x1873576015821312/t0(0) o35->lustre-MDT0000-mdc-ffff9f84513bc800@192.168.202.132@tcp:23/10 lens 392/624 e 0 to 1 dl 1786782158 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 [ 878.588582] Lustre: 18548:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 905.730893] Lustre: DEBUG MARKER: == recovery-small test 13: mdc_readpage restart test (bug 1138) ========================================================== 04:23:04 (1786782184) [ 931.662764] Lustre: DEBUG MARKER: == recovery-small test 14: mdc_readpage resend test (bug 1138) ========================================================== 04:23:30 (1786782210) [ 940.528152] Lustre: DEBUG MARKER: == recovery-small test 15: failed open (-ENOMEM) ========= 04:23:39 (1786782219) [ 950.332773] Lustre: DEBUG MARKER: == recovery-small test 16: timeout bulk put, don't evict client (2732) ========================================================== 04:23:48 (1786782228) [ 998.282946] Lustre: DEBUG MARKER: == recovery-small test 17a: timeout bulk get, don't evict client (2732) ========================================================== 04:24:36 (1786782276) [ 1053.614255] Lustre: DEBUG MARKER: == recovery-small test 17b: timeout bulk get, dont evict client (3582) ========================================================== 04:25:31 (1786782331) [ 1054.923281] Lustre: DEBUG MARKER: SKIP: recovery-small test_17b Needs multiple clients [ 1056.830559] Lustre: DEBUG MARKER: == recovery-small test 18a: manual ost invalidate clears page cache immediately ========================================================== 04:25:35 (1786782335) [ 1057.637576] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1057.718445] Lustre: lustre-OST0001-osc-ffff9f84513bc800: Connection to lustre-OST0001 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1057.734643] Lustre: Skipped 5 previous similar messages [ 1057.749432] LustreError: lustre-OST0001-osc-ffff9f84513bc800: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1064.239906] Lustre: DEBUG MARKER: == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================================================== 04:25:43 (1786782343) [ 1071.093348] LustreError: lustre-OST0000-osc-ffff9f84513bc800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1071.104797] Lustre: lustre-OST0000-osc-ffff9f84513bc800: Connection restored to 192.168.202.132@tcp (at 192.168.202.132@tcp) [ 1071.110905] Lustre: Skipped 4 previous similar messages [ 1095.645988] Lustre: DEBUG MARKER: == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ========================================================== 04:26:14 (1786782374) [ 1099.907433] LustreError: lustre-OST0000-osc-ffff9f84513bc800: operation ost_statfs to node 192.168.202.132@tcp failed: rc = -107 [ 1110.552413] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.202.132@tcp) after server handle changed from 0xa089ad32cb0fc9e5 to 0xa089ad32cb0fcb4a [ 1110.566453] LustreError: lustre-OST0000-osc-ffff9f84513bc800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1121.181338] Lustre: DEBUG MARKER: == recovery-small test 19a: test expired_lock_main on mds (2867) ========================================================== 04:26:39 (1786782399) [ 1121.840221] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1121.843493] Lustre: Skipped 1 previous similar message [ 1139.040235] Lustre: 2389:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786782403/real 1786782403] req@ffff9f8460e3fb80 x1873576015887744/t0(0) o103->lustre-MDT0000-mdc-ffff9f84513bc800@192.168.202.132@tcp:17/18 lens 328/224 e 0 to 1 dl 1786782419 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 [ 1139.088761] Lustre: 2389:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1228.268705] LustreError: lustre-MDT0000-mdc-ffff9f84513bc800: operation ldlm_enqueue to node 192.168.202.132@tcp failed: rc = -107 [ 1228.287210] LustreError: lustre-MDT0000-mdc-ffff9f84513bc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1228.324939] LustreError: 24873:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5 [ 1231.765764] Lustre: Unmounted lustre-client [ 1239.894620] Lustre: DEBUG MARKER: == recovery-small test 19b: test expired_lock_main on ost (2867) ========================================================== 04:28:38 (1786782518) [ 1240.776250] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1349.118033] LustreError: lustre-OST0001-osc-ffff9f84513bc800: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1349.611819] Lustre: Unmounted lustre-client [ 1357.244769] Lustre: DEBUG MARKER: == recovery-small test 19c: check reconnect and lock resend do not trigger expired_lock_main ========================================================== 04:30:35 (1786782635) [ 1357.704701] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1364.746468] Lustre: Unmounted lustre-client [ 1377.768402] Lustre: DEBUG MARKER: == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================================================== 04:30:56 (1786782656) [ 1379.317638] LustreError: lustre-OST0000-osc-ffff9f84513bc800: operation ldlm_enqueue to node 192.168.202.132@tcp failed: rc = -12 [ 1385.905949] Lustre: DEBUG MARKER: == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================================================== 04:31:04 (1786782664) [ 1394.131116] Lustre: DEBUG MARKER: == recovery-small test 21a: drop close request while close and open are both in flight ========================================================== 04:31:12 (1786782672) [ 1424.385790] Lustre: DEBUG MARKER: == recovery-small test 21b: drop open request while close and open are both in flight ========================================================== 04:31:43 (1786782703) [ 1575.650508] Lustre: DEBUG MARKER: == recovery-small test 21c: drop both request while close and open are both in flight ========================================================== 04:34:14 (1786782854) [ 1596.898666] Lustre: lustre-MDT0000-mdc-ffff9f84513bc800: Connection to lustre-MDT0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1596.910684] Lustre: Skipped 18 previous similar messages [ 1596.956563] Lustre: lustre-MDT0000-mdc-ffff9f84513bc800: Connection restored to 192.168.202.132@tcp (at 192.168.202.132@tcp) [ 1596.965084] Lustre: Skipped 16 previous similar messages [ 1605.757254] Lustre: DEBUG MARKER: == recovery-small test 21d: drop close reply while close and open are both in flight ========================================================== 04:34:43 (1786782883) [ 1634.212065] Lustre: DEBUG MARKER: == recovery-small test 21e: drop open reply while close and open are both in flight ========================================================== 04:35:12 (1786782912) [ 1777.634622] Lustre: 30712:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786782915/real 1786782915] req@ffff9f8460e3c380 x1873576016023168/t0(0) o36->lustre-MDT0000-mdc-ffff9f84513bc800@192.168.202.132@tcp:12/10 lens 488/512 e 0 to 1 dl 1786783057 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 1777.709595] Lustre: 30712:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 1780.394436] Lustre: DEBUG MARKER: == recovery-small test 21f: drop both reply while close and open are both in flight ========================================================== 04:37:38 (1786783058) [ 1810.913542] Lustre: DEBUG MARKER: == recovery-small test 21g: drop open reply and close request while close and open are both in flight ========================================================== 04:38:09 (1786783089) [ 1836.362613] Lustre: DEBUG MARKER: == recovery-small test 21h: drop open request and close reply while close and open are both in flight ========================================================== 04:38:35 (1786783115) [ 1862.777832] Lustre: DEBUG MARKER: == recovery-small test 22: drop close request and do mknod ========================================================== 04:39:01 (1786783141) [ 1891.211875] Lustre: DEBUG MARKER: == recovery-small test 23: client hang when close a file after mds crash ========================================================== 04:39:29 (1786783169) [ 1917.216249] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 192.168.202.132@tcp) was lost; in progress operations using this service will fail [ 1927.660638] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0xa089ad32cb0fc045 to 0xa089ad32cb0fe6b7 [ 1943.245794] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1944.672449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1953.484390] Lustre: DEBUG MARKER: == recovery-small test 24a: fsync error (should return error) ========================================================== 04:40:32 (1786783232) [ 1955.318180] LustreError: lustre-OST0000-osc-ffff9f84513bc800: operation ost_write to node 192.168.202.132@tcp failed: rc = -107 [ 1955.326095] LustreError: Skipped 1 previous similar message [ 1955.327886] LustreError: lustre-OST0000-osc-ffff9f84513bc800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1955.357978] Lustre: 2388:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.132@tcp:/lustre/fid: [0x240000402:0x8:0x0]// may get corrupted (rc -5) [ 1955.383851] LustreError: 35258:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9f84513bc800: namespace resource [0x280000400:0x5:0x0].0x0 (ffff9f8447fd5b00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1962.294220] Lustre: DEBUG MARKER: == recovery-small test 24b: test dirty page discard due to client eviction ========================================================== 04:40:40 (1786783240) [ 1964.537439] LustreError: lustre-OST0000-osc-ffff9f84513bc800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1964.551145] Lustre: 2387:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.132@tcp:/lustre/fid: [0x200000405:0x2f:0x0]// may get corrupted (rc -5) [ 1973.804182] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 04:40:52 (1786783252) [ 1977.767205] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node [ 1980.995124] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 04:40:58 (1786783258) [ 1984.817712] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node [ 1988.890548] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 04:41:05 (1786783265) [ 1993.044262] LustreError: lustre-MDT0000-mdc-ffff9f84513bc800: operation mds_close to node 192.168.202.132@tcp failed: rc = -19 [ 1993.061131] LustreError: Skipped 3 previous similar messages [ 2012.128381] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 192.168.202.132@tcp) was lost; in progress operations using this service will fail [ 2022.386559] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0xa089ad32cb0fe6b7 to 0xa089ad32cb10009c [ 2022.418252] BUG: spinlock recursion on CPU#3, ptlrpcd_rcv/2385 [ 2022.424289] lock: 0xffff9f8452a252e0, .magic: dead4ead, .owner: ptlrpcd_rcv/2385, .owner_cpu: 3 [ 2022.433101] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2022.442620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2022.448692] Call Trace: [ 2022.452962] ? dump_stack+0xbb/0x10e [ 2022.457135] ? spin_dump.cold.3+0x32/0x65 [ 2022.460822] ? do_raw_spin_lock+0x10f/0x190 [ 2022.464113] ? _raw_spin_lock+0x21/0x30 [ 2022.469751] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2022.471932] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2022.479660] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2022.481939] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2022.488815] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2022.491450] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2022.493651] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2022.500927] ? __next_timer_interrupt+0x160/0x160 [ 2022.503990] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2022.511492] ? do_wait_intr+0xf0/0xf0 [ 2022.514125] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2022.518079] ? kthread+0x1d1/0x200 [ 2022.520793] ? set_kthread_struct+0x70/0x70 [ 2022.528523] ? ret_from_fork+0x1f/0x30 [ 2048.053464] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2048.057562] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2048.058079] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2048.058086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2048.058198] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2048.058203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2048.060139] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2048.060155] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2048.060229] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2048.060235] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2048.060238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2048.060251] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2048.060255] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2048.060259] RAX: 0000000000007307 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2048.060262] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2048.060264] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2048.060266] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2048.060268] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2048.060272] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2048.060274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2048.060276] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2048.060285] Call Trace: [ 2048.058206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2048.061196] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2048.061200] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2048.061204] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2048.061207] RDX: 00000000000013d7 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2048.061209] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2048.061211] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2048.061213] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2048.061216] [ 2048.061216] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2048.061222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2048.061225] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2048.061225] ? show_regs.cold.9+0x22/0x2f [ 2048.061233] Call Trace: [ 2048.061238] [ 2048.061239] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2048.061244] ? show_regs.cold.9+0x22/0x2f [ 2048.061252] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2048.061255] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2048.061263] ? watchdog+0x40/0x40 [ 2048.061266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2048.061274] ? hrtimer_interrupt+0x128/0x2c0 [ 2048.061276] ? watchdog+0x40/0x40 [ 2048.061287] ? hrtimer_interrupt+0x128/0x2c0 [ 2048.061292] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2048.061299] ? apic_timer_interrupt+0xf/0x20 [ 2048.061303] [ 2048.061306] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2048.061311] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2048.061315] do_raw_spin_lock+0x177/0x190 [ 2048.061320] _raw_spin_lock+0x21/0x30 [ 2048.061325] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2048.061284] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2048.061772] ? apic_timer_interrupt+0xf/0x20 [ 2048.061777] [ 2048.061615] obd_mod_free+0x46/0x140 [ptlrpc] [ 2048.061781] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2048.062324] do_raw_spin_lock+0x177/0x190 [ 2048.062330] _raw_spin_lock+0x21/0x30 [ 2048.062335] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2048.063152] mdc_commit_open+0x91/0xb0 [mdc] [ 2048.063192] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2048.063401] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2048.064754] after_reply+0xe2c/0x1780 [ptlrpc] [ 2048.064975] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2048.065203] ? __next_timer_interrupt+0x160/0x160 [ 2048.065213] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2048.065481] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2048.065682] ? do_raw_spin_unlock+0x75/0x190 [ 2048.065555] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2048.066149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2048.066152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2048.066161] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2048.066165] RSP: 0018:ffffc38541c37500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2048.066170] RAX: 0000000000007a88 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2048.066172] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2048.066174] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2048.066176] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2048.066178] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2048.066182] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2048.066184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2048.066186] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2048.066194] Call Trace: [ 2048.066199] [ 2048.066202] ? show_regs.cold.9+0x22/0x2f [ 2048.066210] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2048.066284] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2048.066289] ? watchdog+0x40/0x40 [ 2048.066363] ? hrtimer_interrupt+0x128/0x2c0 [ 2048.066367] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2048.066375] ? apic_timer_interrupt+0xf/0x20 [ 2048.066379] [ 2048.066381] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2048.066387] do_raw_spin_lock+0x177/0x190 [ 2048.066392] _raw_spin_lock+0x21/0x30 [ 2048.066397] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2048.065691] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2048.067955] ? do_wait_intr+0xf0/0xf0 [ 2048.067966] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2048.068770] mdc_close+0x53b/0xd70 [mdc] [ 2048.068830] lmv_close+0x18d/0x490 [lmv] [ 2048.070096] ? do_raw_spin_unlock+0x75/0x190 [ 2048.069952] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2048.070359] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2048.070439] ll_md_close+0x274/0x800 [lustre] [ 2048.070106] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2048.071834] ? do_wait_intr+0xf0/0xf0 [ 2048.071847] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2048.072100] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2048.072341] ll_file_release+0x128/0xc50 [lustre] [ 2048.072436] __fput+0x10d/0x440 [ 2048.072446] ____fput+0x12/0x20 [ 2048.072451] task_work_run+0xc6/0x110 [ 2048.072459] exit_to_usermode_loop+0x1dd/0x1f0 [ 2048.072467] do_syscall_64+0x3d6/0x440 [ 2048.073414] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2048.073579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic [ 2048.072471] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2048.073637] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [ 2048.073651] RIP: 0033:0x7f1a63983cd8 [ 2048.073657] sha512_generic [ 2048.073666] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2048.073672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2048.073674] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2048.073685] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2048.073689] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2048.073694] RAX: 000000000000158b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2048.073696] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2048.073699] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2048.073701] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2048.073704] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2048.073707] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2048.073709] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2048.073711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2048.073717] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2048.073719] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2048.073727] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2048.073729] Call Trace: [ 2048.073730] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2048.073736] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2048.073737] [ 2048.073740] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2048.073745] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2048.073744] ? show_regs.cold.9+0x22/0x2f [ 2048.073757] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2048.073764] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2048.073769] ? watchdog+0x40/0x40 [ 2048.073774] ? hrtimer_interrupt+0x128/0x2c0 [ 2048.073779] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2048.073787] ? apic_timer_interrupt+0xf/0x20 [ 2048.073792] [ 2048.073795] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2048.073802] do_raw_spin_lock+0x177/0x190 [ 2048.073808] _raw_spin_lock+0x21/0x30 [ 2048.073813] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2048.074310] ? do_raw_spin_unlock+0x75/0x190 [ 2048.074316] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2048.074830] ? do_wait_intr+0xf0/0xf0 [ 2048.074479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2048.075035] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2048.075086] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2048.075116] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2048.075145] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2048.074839] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2048.078865] ? do_wait_intr+0xf0/0xf0 [ 2048.078990] mdc_close+0x53b/0xd70 [mdc] [ 2048.079157] lmv_close+0x18d/0x490 [lmv] [ 2048.079278] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2048.079446] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2048.079527] ll_md_close+0x274/0x800 [lustre] [ 2048.079608] ll_file_release+0x128/0xc50 [lustre] [ 2048.079693] __fput+0x10d/0x440 [ 2048.078884] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2048.079969] kthread+0x1d1/0x200 [ 2048.079975] ? set_kthread_struct+0x70/0x70 [ 2048.079979] ret_from_fork+0x1f/0x30 [ 2048.138494] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2048.138788] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2048.144373] ____fput+0x12/0x20 [ 2048.144397] task_work_run+0xc6/0x110 [ 2048.144406] exit_to_usermode_loop+0x1dd/0x1f0 [ 2048.144414] do_syscall_64+0x3d6/0x440 [ 2048.144482] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2048.144493] RIP: 0033:0x7f1a63983cd8 [ 2048.144502] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2048.144506] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2048.144512] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2048.144514] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2048.144516] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2048.144518] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2048.144521] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2049.421950] lmv_intent_open+0x338/0x1150 [lmv] [ 2049.423706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2049.427476] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2049.430772] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2049.435699] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2049.439824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2049.446512] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2049.450378] ? do_raw_spin_unlock+0x75/0x190 [ 2049.452810] ? _raw_spin_unlock+0x12/0x30 [ 2049.454448] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2049.462501] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2049.465214] ll_atomic_open+0x36a/0xd70 [lustre] [ 2049.467431] lookup_open+0x356/0xb40 [ 2049.469126] do_last+0x313/0xfc0 [ 2049.470247] ? nd_jump_root+0xe5/0x160 [ 2049.471888] ? path_init+0x437/0x520 [ 2049.475842] path_openat+0xf7/0x500 [ 2049.477594] ? do_raw_spin_unlock+0x75/0x190 [ 2049.478887] do_filp_open+0x99/0x140 [ 2049.481133] ? getname_flags+0x6e/0x330 [ 2049.482797] ? __check_object_size+0xff/0x256 [ 2049.489524] ? do_raw_spin_unlock+0x75/0x190 [ 2049.492282] ? _raw_spin_unlock+0x12/0x30 [ 2049.495918] do_sys_openat2+0x2b4/0x410 [ 2049.499415] do_sys_open+0x73/0xa0 [ 2049.501512] __x64_sys_openat+0x24/0x30 [ 2049.503436] do_syscall_64+0xc1/0x440 [ 2049.505391] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2049.507873] RIP: 0033:0x7f1a63983332 [ 2049.513949] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2049.527626] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2049.534293] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2049.542375] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2049.549333] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2049.554213] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2049.559501] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2058.721487] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 35s! [ 2058.736648] Showing busy workqueues and worker pools: [ 2058.738239] workqueue events: flags=0x0 [ 2058.743188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2058.745477] pending: ldlm_pools_recalc_task [ptlrpc] [ 2058.754172] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2058.760246] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4] [ 2058.765602] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2058.767207] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2058.774690] workqueue events_long: flags=0x0 [ 2058.777738] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2058.782766] pending: lu_objects_flush [obdclass] [ 2058.791524] workqueue events_unbound: flags=0x2 [ 2058.793174] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2058.794618] pending: flush_memcg_stats_dwork [ 2058.799622] workqueue events_power_efficient: flags=0x80 [ 2058.801258] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2058.811902] pending: neigh_periodic_work, do_cache_clean [ 2058.813323] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2058.819801] pending: neigh_periodic_work [ 2058.821242] workqueue mm_percpu_wq: flags=0x8 [ 2058.832774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2058.835700] pending: vmstat_update [ 2058.836892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2058.842308] pending: vmstat_update [ 2058.846455] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2058.849975] pending: vmstat_update [ 2058.851220] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2058.855134] pending: vmstat_update [ 2058.860942] workqueue writeback: flags=0x4a [ 2058.862520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2058.873212] pending: wb_workfn, wb_workfn [ 2058.876197] workqueue xprtiod: flags=0xa [ 2058.878274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2058.881808] pending: xs_stream_data_receive_workfn [ 2058.884735] workqueue knbd0-recv: flags=0x1a [ 2058.887085] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2058.891102] in-flight: 647:recv_work [ 2058.893130] workqueue ptlrpc_pinger: flags=0x2 [ 2058.895433] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2058.898589] pending: ptlrpc_pinger_main [ptlrpc] [ 2058.902192] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2048s workers=2 idle: 652 [ 2076.057015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2076.057047] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2076.061229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2076.061308] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2076.061316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2076.061319] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2076.061334] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2076.061339] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2076.061344] RAX: 0000000000004673 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2076.061346] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2076.061348] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2076.061350] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2076.061353] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2076.061356] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2076.061358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2076.061360] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2076.061369] Call Trace: [ 2076.061374] [ 2076.061380] ? show_regs.cold.9+0x22/0x2f [ 2076.061389] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2076.061395] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2076.061400] ? watchdog+0x40/0x40 [ 2076.061403] ? hrtimer_interrupt+0x128/0x2c0 [ 2076.061407] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2076.061415] ? apic_timer_interrupt+0xf/0x20 [ 2076.061419] [ 2076.061422] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2076.061429] do_raw_spin_lock+0x177/0x190 [ 2076.061434] _raw_spin_lock+0x21/0x30 [ 2076.061439] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2076.061920] ? do_raw_spin_unlock+0x75/0x190 [ 2076.061927] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2076.062190] ? do_wait_intr+0xf0/0xf0 [ 2076.062198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2076.062401] mdc_close+0x53b/0xd70 [mdc] [ 2076.062448] lmv_close+0x18d/0x490 [lmv] [ 2076.062479] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2076.062602] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2076.062696] ll_md_close+0x274/0x800 [lustre] [ 2076.062778] ll_file_release+0x128/0xc50 [lustre] [ 2076.062869] __fput+0x10d/0x440 [ 2076.062877] ____fput+0x12/0x20 [ 2076.062881] task_work_run+0xc6/0x110 [ 2076.062888] exit_to_usermode_loop+0x1dd/0x1f0 [ 2076.062894] do_syscall_64+0x3d6/0x440 [ 2076.062899] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2076.062906] RIP: 0033:0x7f1a63983cd8 [ 2076.062913] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2076.062917] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2076.062922] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2076.062924] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2076.062926] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2076.062928] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2076.062930] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2076.064822] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2076.070480] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2076.070555] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2076.070563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2076.070565] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2076.070694] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2076.070699] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2076.070704] RAX: 0000000000007f87 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2076.070706] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2076.070707] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2076.070709] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2076.070711] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2076.070714] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2076.070717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2076.070719] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2076.070728] Call Trace: [ 2076.070734] [ 2076.070740] ? show_regs.cold.9+0x22/0x2f [ 2076.070753] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2076.070761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2076.070765] ? watchdog+0x40/0x40 [ 2076.070769] ? hrtimer_interrupt+0x128/0x2c0 [ 2076.070773] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2076.070780] ? apic_timer_interrupt+0xf/0x20 [ 2076.070803] [ 2076.070807] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2076.070815] do_raw_spin_lock+0x177/0x190 [ 2076.070819] _raw_spin_lock+0x21/0x30 [ 2076.070824] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2076.071100] ? do_raw_spin_unlock+0x75/0x190 [ 2076.071106] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2076.073200] ? do_wait_intr+0xf0/0xf0 [ 2076.073209] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2076.073432] mdc_close+0x53b/0xd70 [mdc] [ 2076.073473] lmv_close+0x18d/0x490 [lmv] [ 2076.073509] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2076.135537] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2076.135680] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2076.135687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2076.135690] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2076.135703] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2076.135707] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2076.135712] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2076.135714] RDX: 0000000000000a68 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2076.135716] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2076.135718] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2076.135720] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2076.135723] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2076.135726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2076.135728] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2076.135736] Call Trace: [ 2076.135742] [ 2076.135746] ? show_regs.cold.9+0x22/0x2f [ 2076.135755] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2076.135761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2076.135765] ? watchdog+0x40/0x40 [ 2076.135769] ? hrtimer_interrupt+0x128/0x2c0 [ 2076.135772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2076.135780] ? apic_timer_interrupt+0xf/0x20 [ 2076.135784] [ 2076.135787] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2076.135950] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2076.135955] do_raw_spin_lock+0x177/0x190 [ 2076.135961] _raw_spin_lock+0x21/0x30 [ 2076.135966] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2076.136230] obd_mod_free+0x46/0x140 [ptlrpc] [ 2076.136556] mdc_commit_open+0x91/0xb0 [mdc] [ 2076.136660] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2076.150711] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2076.151034] ll_md_close+0x274/0x800 [lustre] [ 2076.151117] ll_file_release+0x128/0xc50 [lustre] [ 2076.151718] __fput+0x10d/0x440 [ 2076.151728] ____fput+0x12/0x20 [ 2076.151733] task_work_run+0xc6/0x110 [ 2076.151741] exit_to_usermode_loop+0x1dd/0x1f0 [ 2076.151748] do_syscall_64+0x3d6/0x440 [ 2076.151752] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2076.151760] RIP: 0033:0x7f1a63983cd8 [ 2076.151767] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2076.151771] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2076.151777] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2076.151779] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2076.151781] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2076.151782] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2076.151784] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2077.513540] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2077.524557] after_reply+0xe2c/0x1780 [ptlrpc] [ 2077.537336] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2077.542419] ? __next_timer_interrupt+0x160/0x160 [ 2077.556291] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2077.561156] ? do_wait_intr+0xf0/0xf0 [ 2077.562453] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2077.569488] kthread+0x1d1/0x200 [ 2077.571877] ? set_kthread_struct+0x70/0x70 [ 2077.576959] ret_from_fork+0x1f/0x30 [ 2082.407038] rcu: INFO: rcu_sched self-detected stall on CPU [ 2082.409646] rcu: 3-...!: (32289 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 2082.416376] (t=60010 jiffies g=269633 q=191) [ 2082.418344] rcu: rcu_sched kthread starved for 59658 jiffies! g269633 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2082.428279] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2082.432227] rcu: RCU grace-period kthread stack dump: [ 2082.434032] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2082.447876] Call Trace: [ 2082.448749] __schedule+0x351/0xcb0 [ 2082.449727] schedule+0xc0/0x180 [ 2082.450489] schedule_timeout+0xb4/0x190 [ 2082.458693] ? do_raw_spin_unlock+0x75/0x190 [ 2082.462611] ? __next_timer_interrupt+0x160/0x160 [ 2082.468167] ? prepare_to_swait_event+0xf3/0x140 [ 2082.471515] rcu_gp_kthread+0x8dd/0x15b0 [ 2082.473477] ? force_qs_rnp+0x2c0/0x2c0 [ 2082.475520] kthread+0x1d1/0x200 [ 2082.480086] ? set_kthread_struct+0x70/0x70 [ 2082.481725] ret_from_fork+0x1f/0x30 [ 2082.488164] rcu: Stack dump where RCU GP kthread last ran: [ 2082.494551] Sending NMI from CPU 3 to CPUs 2: [ 2082.498802] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 2082.500041] NMI backtrace for cpu 3 [ 2082.504699] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2082.512539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2082.518512] Call Trace: [ 2082.520546] [ 2082.521505] ? dump_stack+0xbb/0x10e [ 2082.524519] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2082.529255] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2082.532357] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2082.537694] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2082.546460] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2082.549682] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2082.553831] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2082.557270] ? update_process_times+0x6d/0xc0 [ 2082.560533] ? tick_sched_handle.isra.18+0x27/0x90 [ 2082.565659] ? tick_sched_timer+0x53/0xb0 [ 2082.571554] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2082.577435] ? tick_sched_handle.isra.18+0x90/0x90 [ 2082.580360] ? hrtimer_interrupt+0x128/0x2c0 [ 2082.584676] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2082.588521] ? apic_timer_interrupt+0xf/0x20 [ 2082.594437] [ 2082.595324] ? native_safe_halt+0x2e/0x50 [ 2082.597561] ? kvm_wait+0xcd/0xe0 [ 2082.601696] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2082.604675] ? do_raw_spin_lock+0x177/0x190 [ 2082.607691] ? _raw_spin_lock+0x21/0x30 [ 2082.612610] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2082.616959] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2082.621542] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2082.623725] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2082.627921] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2082.633199] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2082.635767] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2082.638801] ? __next_timer_interrupt+0x160/0x160 [ 2082.640353] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2082.648136] ? do_wait_intr+0xf0/0xf0 [ 2082.650203] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2082.652578] ? kthread+0x1d1/0x200 [ 2082.654515] ? set_kthread_struct+0x70/0x70 [ 2082.666843] ? ret_from_fork+0x1f/0x30 [ 2084.053146] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2084.056929] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2084.105524] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2084.114774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2084.119911] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2084.124453] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2084.134501] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2084.139513] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2084.144293] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2084.149428] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2084.151981] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2084.156053] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2084.158641] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2084.171465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2084.178864] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2084.190948] Call Trace: [ 2084.194068] [ 2084.197414] ? show_regs.cold.9+0x22/0x2f [ 2084.205559] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2084.209276] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2084.217965] ? watchdog+0x40/0x40 [ 2084.219506] ? hrtimer_interrupt+0x128/0x2c0 [ 2084.222387] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2084.230947] ? apic_timer_interrupt+0xf/0x20 [ 2084.238323] [ 2084.241659] ? native_safe_halt+0x2e/0x50 [ 2084.249656] kvm_wait+0xcd/0xe0 [ 2084.258445] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2084.269880] do_raw_spin_lock+0x177/0x190 [ 2084.273979] _raw_spin_lock+0x21/0x30 [ 2084.284944] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2084.306901] ? do_raw_spin_unlock+0x75/0x190 [ 2084.311781] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2084.319782] ? do_wait_intr+0xf0/0xf0 [ 2084.333850] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2084.336003] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2084.350371] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2084.362340] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2084.373528] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2084.392175] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2084.394501] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2084.396342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2084.402665] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2084.406103] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2084.411617] lmv_intent_open+0x338/0x1150 [lmv] [ 2084.418477] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2084.421183] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2084.424513] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2084.426826] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2084.429345] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2084.432321] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2084.434749] ? do_raw_spin_unlock+0x75/0x190 [ 2084.437048] ? _raw_spin_unlock+0x12/0x30 [ 2084.439209] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2084.441200] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2084.443404] ll_atomic_open+0x36a/0xd70 [lustre] [ 2084.445869] lookup_open+0x356/0xb40 [ 2084.447313] do_last+0x313/0xfc0 [ 2084.448780] ? nd_jump_root+0xe5/0x160 [ 2084.450340] ? path_init+0x437/0x520 [ 2084.452227] path_openat+0xf7/0x500 [ 2084.453304] ? do_raw_spin_unlock+0x75/0x190 [ 2084.454899] do_filp_open+0x99/0x140 [ 2084.456475] ? getname_flags+0x6e/0x330 [ 2084.458086] ? __check_object_size+0xff/0x256 [ 2084.459815] ? do_raw_spin_unlock+0x75/0x190 [ 2084.461580] ? _raw_spin_unlock+0x12/0x30 [ 2084.463284] do_sys_openat2+0x2b4/0x410 [ 2084.464763] do_sys_open+0x73/0xa0 [ 2084.466135] __x64_sys_openat+0x24/0x30 [ 2084.467713] do_syscall_64+0xc1/0x440 [ 2084.469310] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2084.471241] RIP: 0033:0x7f1a63983332 [ 2084.472539] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2084.479413] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2084.481880] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2084.485293] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2084.488237] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2084.491437] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2084.494404] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2088.930850] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 65s! [ 2088.942493] Showing busy workqueues and worker pools: [ 2088.944552] workqueue events: flags=0x0 [ 2088.946329] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2088.950399] pending: ldlm_pools_recalc_task [ptlrpc] [ 2088.954172] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2088.957223] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4] [ 2088.961311] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2088.965028] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2088.971938] workqueue events_long: flags=0x0 [ 2088.975845] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2088.983304] pending: lu_objects_flush [obdclass] [ 2088.991867] workqueue events_unbound: flags=0x2 [ 2088.996868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2088.999871] pending: flush_memcg_stats_dwork [ 2089.003787] workqueue events_power_efficient: flags=0x80 [ 2089.007917] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2089.011726] pending: neigh_periodic_work, do_cache_clean [ 2089.015628] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.023439] pending: neigh_periodic_work [ 2089.027517] workqueue mm_percpu_wq: flags=0x8 [ 2089.033094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.038348] pending: vmstat_update [ 2089.040085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.043117] pending: vmstat_update [ 2089.044699] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.047994] pending: vmstat_update [ 2089.049920] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.053055] pending: vmstat_update [ 2089.055567] workqueue writeback: flags=0x4a [ 2089.057898] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2089.060784] pending: wb_workfn, wb_workfn [ 2089.062876] workqueue xprtiod: flags=0xa [ 2089.064563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2089.067472] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2089.071139] workqueue knbd0-recv: flags=0x1a [ 2089.073327] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2089.076054] in-flight: 647:recv_work [ 2089.078162] workqueue ptlrpc_pinger: flags=0x2 [ 2089.079987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2089.082414] pending: ptlrpc_pinger_main [ptlrpc] [ 2089.085182] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2079s workers=2 idle: 652 [ 2104.056082] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2104.059914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2104.076216] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2104.080631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2104.083267] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2104.084873] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2104.101655] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2104.111123] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2104.122315] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2104.135228] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2104.146737] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2104.156057] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2104.170379] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2104.178254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2104.182156] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2104.189652] Call Trace: [ 2104.191255] [ 2104.191819] ? show_regs.cold.9+0x22/0x2f [ 2104.195644] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2104.197428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2104.208312] ? watchdog+0x40/0x40 [ 2104.209317] ? hrtimer_interrupt+0x128/0x2c0 [ 2104.210595] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2104.217932] ? apic_timer_interrupt+0xf/0x20 [ 2104.219028] [ 2104.219534] ? native_safe_halt+0x2e/0x50 [ 2104.229592] kvm_wait+0xcd/0xe0 [ 2104.231768] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2104.235692] do_raw_spin_lock+0x177/0x190 [ 2104.238983] _raw_spin_lock+0x21/0x30 [ 2104.247222] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2104.251452] ? do_raw_spin_unlock+0x75/0x190 [ 2104.258717] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2104.266225] ? do_wait_intr+0xf0/0xf0 [ 2104.269672] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2104.274491] mdc_close+0x53b/0xd70 [mdc] [ 2104.282614] lmv_close+0x18d/0x490 [lmv] [ 2104.285655] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2104.299600] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2104.301234] ll_md_close+0x274/0x800 [lustre] [ 2104.308040] ll_file_release+0x128/0xc50 [lustre] [ 2104.312827] __fput+0x10d/0x440 [ 2104.316207] ____fput+0x12/0x20 [ 2104.319886] task_work_run+0xc6/0x110 [ 2104.324367] exit_to_usermode_loop+0x1dd/0x1f0 [ 2104.327015] do_syscall_64+0x3d6/0x440 [ 2104.328976] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2104.334178] RIP: 0033:0x7f1a63983cd8 [ 2104.336763] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2104.355187] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2104.359270] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2104.366970] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2104.374590] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2104.381806] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2104.385148] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2108.057040] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 2108.058010] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 2108.060179] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [ 2108.064678] Modules linked in: [ 2108.064773] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 2108.072878] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2108.072958] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2108.072965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2108.072969] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2108.072984] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2108.072988] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2108.072992] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2108.072995] RDX: 0000000000003bee RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2108.072997] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2108.072999] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2108.073002] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2108.073005] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2108.073008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2108.073010] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2108.073019] Call Trace: [ 2108.073031] [ 2108.073036] ? show_regs.cold.9+0x22/0x2f [ 2108.073045] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2108.073051] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2108.073055] ? watchdog+0x40/0x40 [ 2108.073062] ? hrtimer_interrupt+0x128/0x2c0 [ 2108.073066] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2108.073074] ? apic_timer_interrupt+0xf/0x20 [ 2108.073078] [ 2108.073082] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2108.073086] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2108.073091] do_raw_spin_lock+0x177/0x190 [ 2108.073096] _raw_spin_lock+0x21/0x30 [ 2108.073101] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2108.073533] obd_mod_free+0x46/0x140 [ptlrpc] [ 2108.073843] mdc_commit_open+0x91/0xb0 [mdc] [ 2108.073881] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2108.074105] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2108.074376] after_reply+0xe2c/0x1780 [ptlrpc] [ 2108.074784] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2108.074974] ? __next_timer_interrupt+0x160/0x160 [ 2108.074984] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2108.079614] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 2108.083836] ? do_wait_intr+0xf0/0xf0 [ 2108.083862] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2108.084187] kthread+0x1d1/0x200 [ 2108.084193] ? set_kthread_struct+0x70/0x70 [ 2108.084197] ret_from_fork+0x1f/0x30 [ 2108.350150] intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2108.364414] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2108.377593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2108.388347] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2108.393799] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2108.403553] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2108.408700] RAX: 0000000000002441 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2108.413262] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2108.416730] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2108.424766] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2108.430704] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2108.439070] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2108.444010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2108.447643] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2108.456407] Call Trace: [ 2108.459942] [ 2108.460813] ? show_regs.cold.9+0x22/0x2f [ 2108.462460] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2108.465107] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2108.467407] ? watchdog+0x40/0x40 [ 2108.468855] ? hrtimer_interrupt+0x128/0x2c0 [ 2108.470607] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2108.472570] ? apic_timer_interrupt+0xf/0x20 [ 2108.473911] [ 2108.474444] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2108.477739] do_raw_spin_lock+0x177/0x190 [ 2108.480022] _raw_spin_lock+0x21/0x30 [ 2108.481618] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2108.484195] ? do_raw_spin_unlock+0x75/0x190 [ 2108.486192] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2108.488775] ? do_wait_intr+0xf0/0xf0 [ 2108.490402] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2108.493346] mdc_close+0x53b/0xd70 [mdc] [ 2108.495342] lmv_close+0x18d/0x490 [lmv] [ 2108.497181] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2108.499778] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2108.503762] ll_md_close+0x274/0x800 [lustre] [ 2108.506314] ll_file_release+0x128/0xc50 [lustre] [ 2108.512363] __fput+0x10d/0x440 [ 2108.518200] ____fput+0x12/0x20 [ 2108.520176] task_work_run+0xc6/0x110 [ 2108.521761] exit_to_usermode_loop+0x1dd/0x1f0 [ 2108.523268] do_syscall_64+0x3d6/0x440 [ 2108.524761] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2108.526304] RIP: 0033:0x7f1a63983cd8 [ 2108.528777] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2108.534182] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2108.536765] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2108.539828] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2108.542807] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2108.546156] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2108.550738] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2116.056213] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2116.064651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2116.114107] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2116.119917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2116.131539] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2116.134870] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2116.147194] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2116.154179] RAX: 0000000000007b46 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2116.159225] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2116.163020] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2116.168203] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2116.171658] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2116.177330] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2116.181703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2116.184650] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2116.187958] Call Trace: [ 2116.190185] [ 2116.192316] ? show_regs.cold.9+0x22/0x2f [ 2116.198242] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2116.200068] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2116.207072] ? watchdog+0x40/0x40 [ 2116.210051] ? hrtimer_interrupt+0x128/0x2c0 [ 2116.214133] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2116.219067] ? apic_timer_interrupt+0xf/0x20 [ 2116.221827] [ 2116.222867] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2116.232208] do_raw_spin_lock+0x177/0x190 [ 2116.236033] _raw_spin_lock+0x21/0x30 [ 2116.244935] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2116.253159] ? do_raw_spin_unlock+0x75/0x190 [ 2116.256279] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2116.260895] ? do_wait_intr+0xf0/0xf0 [ 2116.263324] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2116.266393] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2116.269920] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2116.275989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2116.279797] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2116.283986] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2116.287781] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2116.289672] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2116.299039] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2116.302041] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2116.305979] lmv_intent_open+0x338/0x1150 [lmv] [ 2116.307626] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2116.316665] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2116.322799] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2116.329081] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2116.336887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2116.344141] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2116.352196] ? do_raw_spin_unlock+0x75/0x190 [ 2116.356457] ? _raw_spin_unlock+0x12/0x30 [ 2116.363608] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2116.365986] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2116.367605] ll_atomic_open+0x36a/0xd70 [lustre] [ 2116.370121] lookup_open+0x356/0xb40 [ 2116.371014] do_last+0x313/0xfc0 [ 2116.372166] ? nd_jump_root+0xe5/0x160 [ 2116.373133] ? path_init+0x437/0x520 [ 2116.374118] path_openat+0xf7/0x500 [ 2116.375176] ? do_raw_spin_unlock+0x75/0x190 [ 2116.378091] do_filp_open+0x99/0x140 [ 2116.379270] ? getname_flags+0x6e/0x330 [ 2116.383092] ? __check_object_size+0xff/0x256 [ 2116.386561] ? do_raw_spin_unlock+0x75/0x190 [ 2116.389752] ? _raw_spin_unlock+0x12/0x30 [ 2116.391155] do_sys_openat2+0x2b4/0x410 [ 2116.393058] do_sys_open+0x73/0xa0 [ 2116.394308] __x64_sys_openat+0x24/0x30 [ 2116.396829] do_syscall_64+0xc1/0x440 [ 2116.398721] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2116.403529] RIP: 0033:0x7f1a63983332 [ 2116.405730] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2116.414766] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2116.418381] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2116.423231] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2116.427208] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2116.430546] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2116.433698] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2119.175511] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 96s! [ 2119.202234] Showing busy workqueues and worker pools: [ 2119.205315] workqueue events: flags=0x0 [ 2119.208063] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2119.211990] pending: ldlm_pools_recalc_task [ptlrpc] [ 2119.215736] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2119.220007] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2119.243996] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2119.277002] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2119.313967] workqueue events_long: flags=0x0 [ 2119.327119] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2119.342655] pending: lu_objects_flush [obdclass] [ 2119.352615] workqueue events_unbound: flags=0x2 [ 2119.363576] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2119.381623] pending: flush_memcg_stats_dwork [ 2119.398418] workqueue events_power_efficient: flags=0x80 [ 2119.408632] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2119.417129] pending: neigh_periodic_work, do_cache_clean [ 2119.421253] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2119.428396] pending: neigh_periodic_work, check_lifetime [ 2119.438578] workqueue mm_percpu_wq: flags=0x8 [ 2119.447446] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2119.454442] pending: vmstat_update [ 2119.459532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2119.469610] pending: vmstat_update [ 2119.473089] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2119.479170] pending: vmstat_update [ 2119.482411] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2119.486644] pending: vmstat_update [ 2119.490077] workqueue writeback: flags=0x4a [ 2119.494289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2119.504548] pending: wb_workfn, wb_workfn [ 2119.515331] workqueue xprtiod: flags=0xa [ 2119.520243] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2119.528017] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2119.538411] workqueue ipv6_addrconf: flags=0x40008 [ 2119.545861] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2119.563590] pending: addrconf_verify_work [ 2119.572952] workqueue knbd0-recv: flags=0x1a [ 2119.576210] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2119.580910] in-flight: 647:recv_work [ 2119.585426] workqueue ptlrpc_pinger: flags=0x2 [ 2119.592958] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2119.594799] pending: ptlrpc_pinger_main [ptlrpc] [ 2119.608788] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2109s workers=2 idle: 652 [ 2132.057222] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2132.060649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2132.090778] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2132.098275] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2132.103951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2132.106026] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2132.112986] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2132.115793] RAX: 0000000000004088 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2132.118633] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2132.121634] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2132.124284] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2132.127894] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2132.130028] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2132.133047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2132.135610] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2132.138030] Call Trace: [ 2132.139187] [ 2132.140075] ? show_regs.cold.9+0x22/0x2f [ 2132.141455] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2132.143638] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2132.144976] ? watchdog+0x40/0x40 [ 2132.145854] ? hrtimer_interrupt+0x128/0x2c0 [ 2132.147576] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2132.149315] ? apic_timer_interrupt+0xf/0x20 [ 2132.151441] [ 2132.152150] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2132.153769] do_raw_spin_lock+0x177/0x190 [ 2132.155598] _raw_spin_lock+0x21/0x30 [ 2132.156864] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2132.159390] ? do_raw_spin_unlock+0x75/0x190 [ 2132.161159] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2132.163664] ? do_wait_intr+0xf0/0xf0 [ 2132.165472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2132.168267] mdc_close+0x53b/0xd70 [mdc] [ 2132.169493] lmv_close+0x18d/0x490 [lmv] [ 2132.171013] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2132.173552] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2132.175526] ll_md_close+0x274/0x800 [lustre] [ 2132.177658] ll_file_release+0x128/0xc50 [lustre] [ 2132.179395] __fput+0x10d/0x440 [ 2132.180871] ____fput+0x12/0x20 [ 2132.181947] task_work_run+0xc6/0x110 [ 2132.183305] exit_to_usermode_loop+0x1dd/0x1f0 [ 2132.185142] do_syscall_64+0x3d6/0x440 [ 2132.186489] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2132.188961] RIP: 0033:0x7f1a63983cd8 [ 2132.190744] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2132.199170] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2132.202665] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2132.205803] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2132.209110] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2132.212412] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2132.215581] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2136.057027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2136.058012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2136.062614] Modules linked in: lustre(O) [ 2136.067593] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2136.068159] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2136.068167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2136.068170] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2136.068186] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2136.068190] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2136.068195] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2136.068197] RDX: 0000000000003639 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2136.068199] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2136.068202] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2136.068204] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2136.068207] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2136.068211] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2136.068213] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2136.068250] Call Trace: [ 2136.068258] [ 2136.068263] ? show_regs.cold.9+0x22/0x2f [ 2136.068273] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2136.068278] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2136.068283] ? watchdog+0x40/0x40 [ 2136.068288] ? hrtimer_interrupt+0x128/0x2c0 [ 2136.068291] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2136.068299] ? apic_timer_interrupt+0xf/0x20 [ 2136.068304] [ 2136.068307] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2136.068313] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2136.068317] do_raw_spin_lock+0x177/0x190 [ 2136.068384] _raw_spin_lock+0x21/0x30 [ 2136.068390] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2136.068901] obd_mod_free+0x46/0x140 [ptlrpc] [ 2136.069105] mdc_commit_open+0x91/0xb0 [mdc] [ 2136.069141] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2136.069412] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2136.069799] after_reply+0xe2c/0x1780 [ptlrpc] [ 2136.070076] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2136.070293] ? __next_timer_interrupt+0x160/0x160 [ 2136.070303] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2136.070597] ? do_wait_intr+0xf0/0xf0 [ 2136.070608] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2136.073927] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 2136.102157] kthread+0x1d1/0x200 [ 2136.102178] ? set_kthread_struct+0x70/0x70 [ 2136.102183] ret_from_fork+0x1f/0x30 [ 2136.354076] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2136.387725] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2136.402746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2136.415491] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2136.420813] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2136.444886] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2136.448313] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2136.456400] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2136.459381] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2136.464960] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2136.469763] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2136.475712] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2136.480528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2136.483804] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2136.488190] Call Trace: [ 2136.489743] [ 2136.491183] ? show_regs.cold.9+0x22/0x2f [ 2136.494526] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2136.499745] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2136.503698] ? watchdog+0x40/0x40 [ 2136.505853] ? hrtimer_interrupt+0x128/0x2c0 [ 2136.508201] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2136.511720] ? apic_timer_interrupt+0xf/0x20 [ 2136.513157] [ 2136.515763] ? native_safe_halt+0x2e/0x50 [ 2136.518799] kvm_wait+0xcd/0xe0 [ 2136.520584] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2136.523938] do_raw_spin_lock+0x177/0x190 [ 2136.526961] _raw_spin_lock+0x21/0x30 [ 2136.529161] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2136.532985] ? do_raw_spin_unlock+0x75/0x190 [ 2136.536185] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2136.539677] ? do_wait_intr+0xf0/0xf0 [ 2136.541870] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2136.545843] mdc_close+0x53b/0xd70 [mdc] [ 2136.547862] lmv_close+0x18d/0x490 [lmv] [ 2136.549922] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2136.555061] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2136.557734] ll_md_close+0x274/0x800 [lustre] [ 2136.561491] ll_file_release+0x128/0xc50 [lustre] [ 2136.564704] __fput+0x10d/0x440 [ 2136.566655] ____fput+0x12/0x20 [ 2136.568819] task_work_run+0xc6/0x110 [ 2136.570996] exit_to_usermode_loop+0x1dd/0x1f0 [ 2136.573900] do_syscall_64+0x3d6/0x440 [ 2136.576489] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2136.579599] RIP: 0033:0x7f1a63983cd8 [ 2136.582648] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2136.593892] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2136.598811] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2136.603836] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2136.608822] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2136.613514] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2136.618750] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2144.053584] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2144.064742] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2144.164507] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2144.186345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2144.202676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2144.221243] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2144.267090] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2144.277092] RAX: 0000000000002f88 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2144.293070] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2144.309300] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2144.323619] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2144.338433] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2144.350819] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2144.379955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2144.382069] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2144.407093] Call Trace: [ 2144.421374] [ 2144.422237] ? show_regs.cold.9+0x22/0x2f [ 2144.430569] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2144.449290] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2144.450951] ? watchdog+0x40/0x40 [ 2144.463929] ? hrtimer_interrupt+0x128/0x2c0 [ 2144.465603] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2144.482267] ? apic_timer_interrupt+0xf/0x20 [ 2144.492607] [ 2144.495394] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2144.503291] do_raw_spin_lock+0x177/0x190 [ 2144.509468] _raw_spin_lock+0x21/0x30 [ 2144.514382] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2144.518293] ? do_raw_spin_unlock+0x75/0x190 [ 2144.521718] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2144.525355] ? do_wait_intr+0xf0/0xf0 [ 2144.531012] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2144.534743] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2144.544131] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2144.549634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2144.554106] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2144.556618] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2144.563529] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2144.567964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2144.573210] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2144.578391] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2144.580293] lmv_intent_open+0x338/0x1150 [lmv] [ 2144.583364] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2144.587120] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2144.596433] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2144.605677] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2144.609244] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2144.617736] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2144.623374] ? do_raw_spin_unlock+0x75/0x190 [ 2144.630029] ? _raw_spin_unlock+0x12/0x30 [ 2144.633662] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2144.640073] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2144.644021] ll_atomic_open+0x36a/0xd70 [lustre] [ 2144.653073] lookup_open+0x356/0xb40 [ 2144.657347] do_last+0x313/0xfc0 [ 2144.660731] ? nd_jump_root+0xe5/0x160 [ 2144.663491] ? path_init+0x437/0x520 [ 2144.665621] path_openat+0xf7/0x500 [ 2144.668463] ? do_raw_spin_unlock+0x75/0x190 [ 2144.671302] do_filp_open+0x99/0x140 [ 2144.673665] ? getname_flags+0x6e/0x330 [ 2144.679485] ? __check_object_size+0xff/0x256 [ 2144.685206] ? do_raw_spin_unlock+0x75/0x190 [ 2144.690403] ? _raw_spin_unlock+0x12/0x30 [ 2144.693104] do_sys_openat2+0x2b4/0x410 [ 2144.695989] do_sys_open+0x73/0xa0 [ 2144.699774] __x64_sys_openat+0x24/0x30 [ 2144.702290] do_syscall_64+0xc1/0x440 [ 2144.705873] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2144.708707] RIP: 0033:0x7f1a63983332 [ 2144.718141] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2144.739034] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2144.749734] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2144.760041] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2144.768388] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2144.774310] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2144.780054] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2149.862404] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 126s! [ 2149.875023] Showing busy workqueues and worker pools: [ 2149.879564] workqueue events: flags=0x0 [ 2149.881248] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2149.886496] pending: ldlm_pools_recalc_task [ptlrpc] [ 2149.890807] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2149.895654] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2149.902289] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2149.906648] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2149.913408] workqueue events_long: flags=0x0 [ 2149.916429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2149.925356] pending: lu_objects_flush [obdclass] [ 2149.928168] workqueue events_unbound: flags=0x2 [ 2149.933211] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2149.937592] pending: flush_memcg_stats_dwork [ 2149.940589] workqueue events_power_efficient: flags=0x80 [ 2149.944537] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2149.947930] pending: neigh_periodic_work, do_cache_clean [ 2149.950668] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2149.952992] pending: neigh_periodic_work, check_lifetime [ 2149.955879] workqueue mm_percpu_wq: flags=0x8 [ 2149.957933] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2149.961676] pending: vmstat_update [ 2149.963122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2149.966357] pending: vmstat_update [ 2149.968301] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2149.973388] pending: vmstat_update [ 2149.975367] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2149.978542] pending: vmstat_update [ 2149.979914] workqueue writeback: flags=0x4a [ 2149.983367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2149.987826] pending: wb_workfn, wb_workfn [ 2149.990462] workqueue xprtiod: flags=0xa [ 2149.992221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2149.994902] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2149.997581] workqueue ipv6_addrconf: flags=0x40008 [ 2150.001305] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2150.003941] pending: addrconf_verify_work [ 2150.006194] workqueue knbd0-recv: flags=0x1a [ 2150.009219] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2150.012763] in-flight: 647:recv_work [ 2150.015320] workqueue ptlrpc_pinger: flags=0x2 [ 2150.018390] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2150.023504] pending: ptlrpc_pinger_main [ptlrpc] [ 2150.026360] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2140s workers=2 idle: 652 [ 2160.064309] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2160.074502] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2160.134228] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2160.141207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2160.149970] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2160.158812] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2160.204325] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2160.212086] RAX: 0000000000007412 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2160.223924] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2160.233153] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2160.243378] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2160.250523] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2160.259627] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2160.278407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2160.284626] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2160.293478] Call Trace: [ 2160.298379] [ 2160.299333] ? show_regs.cold.9+0x22/0x2f [ 2160.306775] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2160.314091] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2160.328761] ? watchdog+0x40/0x40 [ 2160.332673] ? hrtimer_interrupt+0x128/0x2c0 [ 2160.349601] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2160.355801] ? apic_timer_interrupt+0xf/0x20 [ 2160.360855] [ 2160.366713] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2160.378215] do_raw_spin_lock+0x177/0x190 [ 2160.385628] _raw_spin_lock+0x21/0x30 [ 2160.388374] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2160.402289] ? do_raw_spin_unlock+0x75/0x190 [ 2160.408206] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2160.418111] ? do_wait_intr+0xf0/0xf0 [ 2160.421568] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2160.440059] mdc_close+0x53b/0xd70 [mdc] [ 2160.452046] lmv_close+0x18d/0x490 [lmv] [ 2160.462234] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2160.471546] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2160.481080] ll_md_close+0x274/0x800 [lustre] [ 2160.487759] ll_file_release+0x128/0xc50 [lustre] [ 2160.494657] __fput+0x10d/0x440 [ 2160.503020] ____fput+0x12/0x20 [ 2160.507181] task_work_run+0xc6/0x110 [ 2160.519846] exit_to_usermode_loop+0x1dd/0x1f0 [ 2160.521535] do_syscall_64+0x3d6/0x440 [ 2160.533623] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2160.548639] RIP: 0033:0x7f1a63983cd8 [ 2160.551370] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2160.581454] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2160.593267] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2160.603228] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2160.616739] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2160.630922] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2160.640578] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2164.058764] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2164.065599] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2164.074919] Modules linked in: lustre(O) osp(O) [ 2164.087688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2164.087820] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2164.087828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2164.087831] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2164.087844] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2164.087849] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2164.087854] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2164.087856] RDX: 0000000000007f0b RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2164.087858] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2164.087860] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2164.087863] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2164.087866] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2164.087870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2164.087872] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2164.087881] Call Trace: [ 2164.087886] [ 2164.087893] ? show_regs.cold.9+0x22/0x2f [ 2164.087904] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2164.087911] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2164.087916] ? watchdog+0x40/0x40 [ 2164.087920] ? hrtimer_interrupt+0x128/0x2c0 [ 2164.087924] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2164.087932] ? apic_timer_interrupt+0xf/0x20 [ 2164.087936] [ 2164.087939] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2164.088512] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2164.088519] do_raw_spin_lock+0x177/0x190 [ 2164.088524] _raw_spin_lock+0x21/0x30 [ 2164.088529] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2164.089606] obd_mod_free+0x46/0x140 [ptlrpc] [ 2164.107164] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2164.107236] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2164.107245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2164.107247] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2164.107260] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2164.107264] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2164.107268] RAX: 000000000000140f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2164.107270] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2164.107272] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2164.107274] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2164.107276] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2164.107279] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2164.107282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2164.107284] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2164.107292] Call Trace: [ 2164.107297] [ 2164.107303] ? show_regs.cold.9+0x22/0x2f [ 2164.107312] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2164.107318] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2164.107323] ? watchdog+0x40/0x40 [ 2164.107327] ? hrtimer_interrupt+0x128/0x2c0 [ 2164.107331] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2164.107339] ? apic_timer_interrupt+0xf/0x20 [ 2164.107344] [ 2164.107347] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2164.107353] do_raw_spin_lock+0x177/0x190 [ 2164.107358] _raw_spin_lock+0x21/0x30 [ 2164.107363] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2164.107629] ? do_raw_spin_unlock+0x75/0x190 [ 2164.107635] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2164.107864] ? do_wait_intr+0xf0/0xf0 [ 2164.107872] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2164.181504] mdc_commit_open+0x91/0xb0 [mdc] [ 2164.181560] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2164.181844] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2164.182050] after_reply+0xe2c/0x1780 [ptlrpc] [ 2164.182244] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2164.182441] ? __next_timer_interrupt+0x160/0x160 [ 2164.182451] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2164.200931] mdc_close+0x53b/0xd70 [mdc] [ 2164.200994] lmv_close+0x18d/0x490 [lmv] [ 2164.201023] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2164.201131] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2164.201210] ll_md_close+0x274/0x800 [lustre] [ 2164.201292] ll_file_release+0x128/0xc50 [lustre] [ 2164.201373] __fput+0x10d/0x440 [ 2164.201382] ____fput+0x12/0x20 [ 2164.201387] task_work_run+0xc6/0x110 [ 2164.201394] exit_to_usermode_loop+0x1dd/0x1f0 [ 2164.201401] do_syscall_64+0x3d6/0x440 [ 2164.201406] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2164.201414] RIP: 0033:0x7f1a63983cd8 [ 2164.201457] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2164.201463] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2164.201468] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2164.201470] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2164.201473] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2164.201474] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2164.201476] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2164.883085] ? do_wait_intr+0xf0/0xf0 [ 2164.890846] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2164.903640] kthread+0x1d1/0x200 [ 2164.910403] ? set_kthread_struct+0x70/0x70 [ 2164.916261] ret_from_fork+0x1f/0x30 [ 2176.054908] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2176.060524] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2176.105071] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2176.113973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2176.119412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.126040] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2176.135399] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2176.142782] RAX: 0000000000005283 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2176.150026] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2176.159766] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2176.165919] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2176.169949] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2176.181904] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2176.190211] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2176.193744] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2176.201326] Call Trace: [ 2176.202506] [ 2176.204410] ? show_regs.cold.9+0x22/0x2f [ 2176.211004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2176.214337] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2176.219577] ? watchdog+0x40/0x40 [ 2176.221921] ? hrtimer_interrupt+0x128/0x2c0 [ 2176.226378] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2176.230323] ? apic_timer_interrupt+0xf/0x20 [ 2176.231725] [ 2176.235135] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.237413] do_raw_spin_lock+0x177/0x190 [ 2176.243062] _raw_spin_lock+0x21/0x30 [ 2176.247967] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2176.250545] ? do_raw_spin_unlock+0x75/0x190 [ 2176.255910] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2176.261899] ? do_wait_intr+0xf0/0xf0 [ 2176.263489] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2176.270060] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2176.272438] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2176.278266] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2176.280555] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2176.283015] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2176.285202] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2176.287136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2176.289649] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2176.292416] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2176.294945] lmv_intent_open+0x338/0x1150 [lmv] [ 2176.296800] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2176.299289] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2176.301561] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2176.303482] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2176.305778] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2176.308305] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2176.310528] ? do_raw_spin_unlock+0x75/0x190 [ 2176.312296] ? _raw_spin_unlock+0x12/0x30 [ 2176.314134] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2176.316379] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2176.318587] ll_atomic_open+0x36a/0xd70 [lustre] [ 2176.320971] lookup_open+0x356/0xb40 [ 2176.322525] do_last+0x313/0xfc0 [ 2176.324105] ? nd_jump_root+0xe5/0x160 [ 2176.325754] ? path_init+0x437/0x520 [ 2176.327439] path_openat+0xf7/0x500 [ 2176.329064] ? do_raw_spin_unlock+0x75/0x190 [ 2176.330924] do_filp_open+0x99/0x140 [ 2176.332319] ? getname_flags+0x6e/0x330 [ 2176.334038] ? __check_object_size+0xff/0x256 [ 2176.336254] ? do_raw_spin_unlock+0x75/0x190 [ 2176.338104] ? _raw_spin_unlock+0x12/0x30 [ 2176.340192] do_sys_openat2+0x2b4/0x410 [ 2176.342510] do_sys_open+0x73/0xa0 [ 2176.344339] __x64_sys_openat+0x24/0x30 [ 2176.346006] do_syscall_64+0xc1/0x440 [ 2176.347535] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2176.349798] RIP: 0033:0x7f1a63983332 [ 2176.351474] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2176.360440] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2176.364070] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2176.367177] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2176.373216] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2176.376700] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2176.380199] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2180.067106] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 157s! [ 2180.071549] Showing busy workqueues and worker pools: [ 2180.074339] workqueue events: flags=0x0 [ 2180.079028] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2180.084005] pending: ldlm_pools_recalc_task [ptlrpc] [ 2180.089702] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2180.094609] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2180.100895] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2180.105270] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2180.109824] workqueue events_long: flags=0x0 [ 2180.111849] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2180.114927] pending: lu_objects_flush [obdclass] [ 2180.119330] workqueue events_unbound: flags=0x2 [ 2180.120712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2180.124486] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2180.128475] workqueue events_power_efficient: flags=0x80 [ 2180.130393] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2180.135373] pending: neigh_periodic_work, do_cache_clean [ 2180.138302] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2180.141742] pending: neigh_periodic_work, check_lifetime [ 2180.144607] workqueue mm_percpu_wq: flags=0x8 [ 2180.146485] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2180.149319] pending: vmstat_update [ 2180.151077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2180.154036] pending: vmstat_update [ 2180.155720] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2180.158702] pending: vmstat_update [ 2180.160261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2180.163474] pending: vmstat_update [ 2180.165459] workqueue writeback: flags=0x4a [ 2180.168066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2180.171671] pending: wb_workfn, wb_workfn [ 2180.173978] workqueue xprtiod: flags=0xa [ 2180.175906] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2180.178778] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2180.181555] workqueue ipv6_addrconf: flags=0x40008 [ 2180.183606] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2180.186486] pending: addrconf_verify_work [ 2180.188495] workqueue knbd0-recv: flags=0x1a [ 2180.190463] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2180.193522] in-flight: 647:recv_work [ 2180.195606] workqueue ptlrpc_pinger: flags=0x2 [ 2180.197701] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2180.200799] pending: ptlrpc_pinger_main [ptlrpc] [ 2180.203470] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2170s workers=2 idle: 652 [ 2188.056126] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2188.061811] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2188.095957] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2188.111781] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2188.115900] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2188.118444] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2188.135667] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2188.139964] RAX: 0000000000005483 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2188.143427] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2188.146647] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2188.150938] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2188.158481] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2188.162298] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2188.166307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2188.168196] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2188.171577] Call Trace: [ 2188.179520] [ 2188.180824] ? show_regs.cold.9+0x22/0x2f [ 2188.186441] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2188.189252] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2188.192225] ? watchdog+0x40/0x40 [ 2188.195075] ? hrtimer_interrupt+0x128/0x2c0 [ 2188.197440] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2188.200755] ? apic_timer_interrupt+0xf/0x20 [ 2188.203187] [ 2188.204370] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2188.210545] do_raw_spin_lock+0x177/0x190 [ 2188.211920] _raw_spin_lock+0x21/0x30 [ 2188.212964] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2188.220460] ? do_raw_spin_unlock+0x75/0x190 [ 2188.227506] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2188.235346] ? do_wait_intr+0xf0/0xf0 [ 2188.236986] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2188.242160] mdc_close+0x53b/0xd70 [mdc] [ 2188.243504] lmv_close+0x18d/0x490 [lmv] [ 2188.253011] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2188.255227] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2188.260485] ll_md_close+0x274/0x800 [lustre] [ 2188.262345] ll_file_release+0x128/0xc50 [lustre] [ 2188.264925] __fput+0x10d/0x440 [ 2188.269086] ____fput+0x12/0x20 [ 2188.273367] task_work_run+0xc6/0x110 [ 2188.274626] exit_to_usermode_loop+0x1dd/0x1f0 [ 2188.278339] do_syscall_64+0x3d6/0x440 [ 2188.280106] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2188.285153] RIP: 0033:0x7f1a63983cd8 [ 2188.286079] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2188.310228] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2188.316606] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2188.328587] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2188.331513] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2188.340894] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2188.347903] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2192.057548] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2192.059757] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2192.060929] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2192.061092] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2192.061104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2192.061107] RIP: 0010:__pv_queued_spin_lock_slowpath+0x2fa/0x6d0 [ 2192.061127] Code: 5b d4 dc 02 01 49 c1 e5 05 48 98 49 81 c5 40 41 03 00 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 <48> 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f [ 2192.061132] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [ 2192.061140] RAX: 0000000000005300 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2192.061142] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2192.061144] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2192.061150] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2192.061152] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2192.061155] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2192.061334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2192.061337] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2192.061344] Call Trace: [ 2192.061378] [ 2192.061466] ? show_regs.cold.9+0x22/0x2f [ 2192.061478] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2192.061486] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2192.061491] ? watchdog+0x40/0x40 [ 2192.061495] ? hrtimer_interrupt+0x128/0x2c0 [ 2192.061499] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2192.061507] ? apic_timer_interrupt+0xf/0x20 [ 2192.061703] [ 2192.065831] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2192.065911] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2192.065930] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2192.065933] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2192.065954] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2192.065959] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2192.065962] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2192.065964] RDX: 0000000000004b76 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2192.065967] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2192.065969] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2192.065971] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2192.065974] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2192.065977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2192.065979] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2192.065985] Call Trace: [ 2192.065990] [ 2192.065996] ? show_regs.cold.9+0x22/0x2f [ 2192.066006] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2192.066013] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2192.066017] ? watchdog+0x40/0x40 [ 2192.066021] ? hrtimer_interrupt+0x128/0x2c0 [ 2192.066024] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2192.066033] ? apic_timer_interrupt+0xf/0x20 [ 2192.066037] [ 2192.066040] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2192.066044] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2192.066048] do_raw_spin_lock+0x177/0x190 [ 2192.066052] _raw_spin_lock+0x21/0x30 [ 2192.066057] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2192.066326] obd_mod_free+0x46/0x140 [ptlrpc] [ 2192.066786] mdc_commit_open+0x91/0xb0 [mdc] [ 2192.066820] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2192.067088] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2192.067300] after_reply+0xe2c/0x1780 [ptlrpc] [ 2192.067704] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2192.067897] ? __next_timer_interrupt+0x160/0x160 [ 2192.067906] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2192.089066] ? __pv_queued_spin_lock_slowpath+0x2fa/0x6d0 [ 2192.094214] ? do_wait_intr+0xf0/0xf0 [ 2192.099136] do_raw_spin_lock+0x177/0x190 [ 2192.102066] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2192.116820] _raw_spin_lock+0x21/0x30 [ 2192.120048] kthread+0x1d1/0x200 [ 2192.120065] ? set_kthread_struct+0x70/0x70 [ 2192.120073] ret_from_fork+0x1f/0x30 [ 2192.124720] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2192.384809] ? do_raw_spin_unlock+0x75/0x190 [ 2192.388844] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2192.392142] ? do_wait_intr+0xf0/0xf0 [ 2192.397041] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2192.403841] mdc_close+0x53b/0xd70 [mdc] [ 2192.411515] lmv_close+0x18d/0x490 [lmv] [ 2192.416166] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2192.423332] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2192.432510] ll_md_close+0x274/0x800 [lustre] [ 2192.438894] ll_file_release+0x128/0xc50 [lustre] [ 2192.444536] __fput+0x10d/0x440 [ 2192.449068] ____fput+0x12/0x20 [ 2192.454235] task_work_run+0xc6/0x110 [ 2192.460115] exit_to_usermode_loop+0x1dd/0x1f0 [ 2192.467081] do_syscall_64+0x3d6/0x440 [ 2192.471779] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2192.476407] RIP: 0033:0x7f1a63983cd8 [ 2192.480611] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2192.501848] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2192.509776] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2192.516690] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2192.523079] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2192.539106] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2192.547296] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2208.054793] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2208.058802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2208.112105] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2208.124285] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2208.132754] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2208.137686] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2208.151134] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2208.156683] RAX: 0000000000004e86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2208.162857] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2208.166692] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2208.172727] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2208.176390] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2208.179696] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2208.186648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2208.195727] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2208.201420] Call Trace: [ 2208.202342] [ 2208.203009] ? show_regs.cold.9+0x22/0x2f [ 2208.204783] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2208.206700] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2208.208720] ? watchdog+0x40/0x40 [ 2208.210120] ? hrtimer_interrupt+0x128/0x2c0 [ 2208.212394] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2208.214921] ? apic_timer_interrupt+0xf/0x20 [ 2208.217495] [ 2208.219075] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2208.221362] do_raw_spin_lock+0x177/0x190 [ 2208.227459] _raw_spin_lock+0x21/0x30 [ 2208.234958] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2208.241947] ? do_raw_spin_unlock+0x75/0x190 [ 2208.248246] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2208.256451] ? do_wait_intr+0xf0/0xf0 [ 2208.262362] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2208.267786] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2208.270757] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2208.281094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2208.295468] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2208.302403] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2208.310583] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2208.320527] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2208.324156] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2208.326697] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2208.335494] lmv_intent_open+0x338/0x1150 [lmv] [ 2208.345681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2208.347618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2208.355111] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2208.363475] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2208.371491] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2208.374560] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2208.382797] ? do_raw_spin_unlock+0x75/0x190 [ 2208.388129] ? _raw_spin_unlock+0x12/0x30 [ 2208.396760] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2208.403751] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2208.409184] ll_atomic_open+0x36a/0xd70 [lustre] [ 2208.418768] lookup_open+0x356/0xb40 [ 2208.426869] do_last+0x313/0xfc0 [ 2208.434454] ? nd_jump_root+0xe5/0x160 [ 2208.438800] ? path_init+0x437/0x520 [ 2208.445923] path_openat+0xf7/0x500 [ 2208.447880] ? do_raw_spin_unlock+0x75/0x190 [ 2208.464682] do_filp_open+0x99/0x140 [ 2208.465849] ? getname_flags+0x6e/0x330 [ 2208.466911] ? __check_object_size+0xff/0x256 [ 2208.473544] ? do_raw_spin_unlock+0x75/0x190 [ 2208.475103] ? _raw_spin_unlock+0x12/0x30 [ 2208.478627] do_sys_openat2+0x2b4/0x410 [ 2208.481442] do_sys_open+0x73/0xa0 [ 2208.485252] __x64_sys_openat+0x24/0x30 [ 2208.488698] do_syscall_64+0xc1/0x440 [ 2208.491448] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2208.498291] RIP: 0033:0x7f1a63983332 [ 2208.499580] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2208.515737] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2208.532188] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2208.543939] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2208.553361] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2208.562776] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2208.570626] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2210.272069] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 187s! [ 2210.276168] Showing busy workqueues and worker pools: [ 2210.278993] workqueue events: flags=0x0 [ 2210.281790] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2210.285159] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2210.290301] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2210.295787] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2210.302680] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2210.306282] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2210.311534] workqueue events_long: flags=0x0 [ 2210.315174] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2210.319222] pending: lu_objects_flush [obdclass] [ 2210.322748] workqueue events_unbound: flags=0x2 [ 2210.325797] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2210.329236] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2210.332325] workqueue events_power_efficient: flags=0x80 [ 2210.335161] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2210.338426] pending: neigh_periodic_work, do_cache_clean [ 2210.341104] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2210.345445] pending: neigh_periodic_work, check_lifetime [ 2210.347671] workqueue mm_percpu_wq: flags=0x8 [ 2210.350255] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2210.353267] pending: vmstat_update [ 2210.355287] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2210.358593] pending: vmstat_update [ 2210.361217] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2210.366660] pending: vmstat_update [ 2210.368217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2210.371185] pending: vmstat_update [ 2210.372956] workqueue writeback: flags=0x4a [ 2210.375349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2210.378528] pending: wb_workfn, wb_workfn [ 2210.380426] workqueue xprtiod: flags=0xa [ 2210.383400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2210.386396] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2210.389018] workqueue ipv6_addrconf: flags=0x40008 [ 2210.391263] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2210.394391] pending: addrconf_verify_work [ 2210.396710] workqueue knbd0-recv: flags=0x1a [ 2210.398725] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2210.401654] in-flight: 647:recv_work [ 2210.403631] workqueue ptlrpc_pinger: flags=0x2 [ 2210.405564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2210.408301] pending: ptlrpc_pinger_main [ptlrpc] [ 2210.410987] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2200s workers=2 idle: 652 [ 2216.058713] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2216.066881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2216.122009] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2216.131818] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2216.139313] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2216.144854] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2216.153961] RSP: 0018:ffffc38542073af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2216.158050] RAX: 000000000000768f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2216.161573] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2216.165411] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2216.168772] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2216.175223] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2216.182064] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2216.189567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2216.193707] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2216.197617] Call Trace: [ 2216.198761] [ 2216.199731] ? show_regs.cold.9+0x22/0x2f [ 2216.201769] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2216.204081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2216.206996] ? watchdog+0x40/0x40 [ 2216.208186] ? hrtimer_interrupt+0x128/0x2c0 [ 2216.212884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2216.219029] ? apic_timer_interrupt+0xf/0x20 [ 2216.227798] [ 2216.229865] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2216.237892] do_raw_spin_lock+0x177/0x190 [ 2216.239371] _raw_spin_lock+0x21/0x30 [ 2216.250061] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2216.259204] ? do_raw_spin_unlock+0x75/0x190 [ 2216.264723] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2216.271099] ? do_wait_intr+0xf0/0xf0 [ 2216.273741] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2216.279158] mdc_close+0x53b/0xd70 [mdc] [ 2216.287762] lmv_close+0x18d/0x490 [lmv] [ 2216.292989] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2216.304209] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2216.311939] ll_md_close+0x274/0x800 [lustre] [ 2216.314615] ll_file_release+0x128/0xc50 [lustre] [ 2216.320922] __fput+0x10d/0x440 [ 2216.322273] ____fput+0x12/0x20 [ 2216.324777] task_work_run+0xc6/0x110 [ 2216.329039] exit_to_usermode_loop+0x1dd/0x1f0 [ 2216.332633] do_syscall_64+0x3d6/0x440 [ 2216.337160] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2216.340284] RIP: 0033:0x7f1a63983cd8 [ 2216.343625] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2216.363185] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2216.371413] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2216.376384] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2216.380086] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2216.384038] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2216.387938] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2220.057063] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2220.059279] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2220.064447] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2220.100723] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2220.100736] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2220.100739] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2220.100752] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2220.100756] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2220.100762] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2220.100764] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2220.100766] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2220.100768] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2220.100770] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2220.100773] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2220.100776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2220.100778] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2220.100784] Call Trace: [ 2220.100789] [ 2220.100793] ? show_regs.cold.9+0x22/0x2f [ 2220.100802] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2220.100807] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2220.100812] ? watchdog+0x40/0x40 [ 2220.100816] ? hrtimer_interrupt+0x128/0x2c0 [ 2220.100820] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2220.100826] ? apic_timer_interrupt+0xf/0x20 [ 2220.100830] [ 2220.100832] ? native_safe_halt+0x2e/0x50 [ 2220.100836] kvm_wait+0xcd/0xe0 [ 2220.100842] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2220.100851] do_raw_spin_lock+0x177/0x190 [ 2220.100855] _raw_spin_lock+0x21/0x30 [ 2220.100859] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2220.101136] ? do_raw_spin_unlock+0x75/0x190 [ 2220.101142] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2220.108317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2220.108669] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2220.108676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2220.108679] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2220.108694] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2220.108698] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2220.108703] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2220.108706] RDX: 00000000000017f1 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2220.108708] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2220.108710] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2220.108713] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2220.108716] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2220.108719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2220.108722] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2220.108731] Call Trace: [ 2220.108736] [ 2220.108742] ? show_regs.cold.9+0x22/0x2f [ 2220.108751] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2220.108976] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2220.108981] ? watchdog+0x40/0x40 [ 2220.108985] ? hrtimer_interrupt+0x128/0x2c0 [ 2220.108990] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2220.109000] ? apic_timer_interrupt+0xf/0x20 [ 2220.109004] [ 2220.109008] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2220.109013] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2220.109018] do_raw_spin_lock+0x177/0x190 [ 2220.109023] _raw_spin_lock+0x21/0x30 [ 2220.109028] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2220.109914] obd_mod_free+0x46/0x140 [ptlrpc] [ 2220.110107] mdc_commit_open+0x91/0xb0 [mdc] [ 2220.110279] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2220.110604] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2220.110809] after_reply+0xe2c/0x1780 [ptlrpc] [ 2220.118264] ? do_wait_intr+0xf0/0xf0 [ 2220.118286] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2220.128138] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2220.128441] ? __next_timer_interrupt+0x160/0x160 [ 2220.128953] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2220.129176] ? do_wait_intr+0xf0/0xf0 [ 2220.129186] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2220.129381] kthread+0x1d1/0x200 [ 2220.129385] ? set_kthread_struct+0x70/0x70 [ 2220.129442] ret_from_fork+0x1f/0x30 [ 2220.393075] mdc_close+0x53b/0xd70 [mdc] [ 2220.394876] lmv_close+0x18d/0x490 [lmv] [ 2220.396585] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2220.402606] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2220.404632] ll_md_close+0x274/0x800 [lustre] [ 2220.409756] ll_file_release+0x128/0xc50 [lustre] [ 2220.411194] __fput+0x10d/0x440 [ 2220.413847] ____fput+0x12/0x20 [ 2220.415208] task_work_run+0xc6/0x110 [ 2220.423802] exit_to_usermode_loop+0x1dd/0x1f0 [ 2220.427782] do_syscall_64+0x3d6/0x440 [ 2220.430251] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2220.433256] RIP: 0033:0x7f1a63983cd8 [ 2220.435812] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2220.453694] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2220.457102] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2220.466962] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2220.472764] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2220.477138] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2220.479772] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2236.053033] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2236.055582] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2236.087695] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2236.095985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2236.101773] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2236.104276] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2236.118999] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2236.124415] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2236.131412] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2236.134469] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2236.137837] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2236.141744] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2236.151650] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2236.161351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2236.165998] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2236.171511] Call Trace: [ 2236.172890] [ 2236.174020] ? show_regs.cold.9+0x22/0x2f [ 2236.176038] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2236.178953] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2236.181470] ? watchdog+0x40/0x40 [ 2236.182760] ? hrtimer_interrupt+0x128/0x2c0 [ 2236.184067] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2236.187516] ? apic_timer_interrupt+0xf/0x20 [ 2236.194219] [ 2236.195800] ? native_safe_halt+0x2e/0x50 [ 2236.199059] kvm_wait+0xcd/0xe0 [ 2236.200613] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2236.202976] do_raw_spin_lock+0x177/0x190 [ 2236.204743] _raw_spin_lock+0x21/0x30 [ 2236.205891] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2236.207951] ? do_raw_spin_unlock+0x75/0x190 [ 2236.209732] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2236.213497] ? do_wait_intr+0xf0/0xf0 [ 2236.216081] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2236.219761] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2236.223315] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2236.230143] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2236.233421] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2236.236781] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2236.241737] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2236.244924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2236.251097] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2236.254798] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2236.260762] lmv_intent_open+0x338/0x1150 [lmv] [ 2236.262475] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2236.266221] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2236.275209] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2236.277090] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2236.282710] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2236.290806] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2236.296373] ? do_raw_spin_unlock+0x75/0x190 [ 2236.298607] ? _raw_spin_unlock+0x12/0x30 [ 2236.300649] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2236.302573] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2236.308488] ll_atomic_open+0x36a/0xd70 [lustre] [ 2236.312623] lookup_open+0x356/0xb40 [ 2236.317824] do_last+0x313/0xfc0 [ 2236.320569] ? nd_jump_root+0xe5/0x160 [ 2236.322200] ? path_init+0x437/0x520 [ 2236.324766] path_openat+0xf7/0x500 [ 2236.326075] ? do_raw_spin_unlock+0x75/0x190 [ 2236.328701] do_filp_open+0x99/0x140 [ 2236.330478] ? getname_flags+0x6e/0x330 [ 2236.332099] ? __check_object_size+0xff/0x256 [ 2236.334051] ? do_raw_spin_unlock+0x75/0x190 [ 2236.335886] ? _raw_spin_unlock+0x12/0x30 [ 2236.339702] do_sys_openat2+0x2b4/0x410 [ 2236.341195] do_sys_open+0x73/0xa0 [ 2236.342752] __x64_sys_openat+0x24/0x30 [ 2236.344658] do_syscall_64+0xc1/0x440 [ 2236.347546] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2236.349595] RIP: 0033:0x7f1a63983332 [ 2236.351092] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2236.369695] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2236.373030] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2236.375520] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2236.379455] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2236.390084] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2236.393852] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2240.482077] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 217s! [ 2240.485281] Showing busy workqueues and worker pools: [ 2240.487634] workqueue events: flags=0x0 [ 2240.489269] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2240.502815] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2240.511146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2240.519918] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2240.525022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2240.528814] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2240.537137] workqueue events_long: flags=0x0 [ 2240.540732] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2240.542755] pending: lu_objects_flush [obdclass] [ 2240.544833] workqueue events_unbound: flags=0x2 [ 2240.547137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2240.550445] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2240.553504] workqueue events_power_efficient: flags=0x80 [ 2240.555293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2240.559964] pending: neigh_periodic_work, do_cache_clean [ 2240.565101] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2240.570389] pending: neigh_periodic_work, check_lifetime [ 2240.573139] workqueue mm_percpu_wq: flags=0x8 [ 2240.577384] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2240.580964] pending: vmstat_update [ 2240.585600] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2240.589963] pending: vmstat_update [ 2240.593105] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2240.600586] pending: vmstat_update [ 2240.602095] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2240.615822] pending: vmstat_update [ 2240.617901] workqueue writeback: flags=0x4a [ 2240.619762] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2240.622667] pending: wb_workfn, wb_workfn [ 2240.625382] workqueue xprtiod: flags=0xa [ 2240.627181] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2240.629534] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2240.631599] workqueue ipv6_addrconf: flags=0x40008 [ 2240.635020] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2240.639972] pending: addrconf_verify_work [ 2240.643912] workqueue knbd0-recv: flags=0x1a [ 2240.646886] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2240.650936] in-flight: 647:recv_work [ 2240.655535] workqueue ptlrpc_pinger: flags=0x2 [ 2240.657678] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2240.660891] pending: ptlrpc_pinger_main [ptlrpc] [ 2240.664817] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2230s workers=2 idle: 652 [ 2244.056945] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2244.066986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2244.097456] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2244.103374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2244.109959] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2244.112177] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2244.126904] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2244.130061] RAX: 0000000000007689 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2244.133259] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2244.142205] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2244.149481] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2244.153064] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2244.158039] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2244.167018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2244.169918] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2244.173518] Call Trace: [ 2244.176253] [ 2244.177027] ? show_regs.cold.9+0x22/0x2f [ 2244.179591] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2244.182912] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2244.184843] ? watchdog+0x40/0x40 [ 2244.189068] ? hrtimer_interrupt+0x128/0x2c0 [ 2244.192134] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2244.198459] ? apic_timer_interrupt+0xf/0x20 [ 2244.200057] [ 2244.201543] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2244.205866] do_raw_spin_lock+0x177/0x190 [ 2244.209190] _raw_spin_lock+0x21/0x30 [ 2244.210927] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2244.213761] ? do_raw_spin_unlock+0x75/0x190 [ 2244.215885] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2244.219818] ? do_wait_intr+0xf0/0xf0 [ 2244.221846] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2244.224418] mdc_close+0x53b/0xd70 [mdc] [ 2244.227682] lmv_close+0x18d/0x490 [lmv] [ 2244.229515] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2244.239107] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2244.244319] ll_md_close+0x274/0x800 [lustre] [ 2244.247755] ll_file_release+0x128/0xc50 [lustre] [ 2244.250504] __fput+0x10d/0x440 [ 2244.251632] ____fput+0x12/0x20 [ 2244.255363] task_work_run+0xc6/0x110 [ 2244.257829] exit_to_usermode_loop+0x1dd/0x1f0 [ 2244.261722] do_syscall_64+0x3d6/0x440 [ 2244.263802] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2244.267461] RIP: 0033:0x7f1a63983cd8 [ 2244.268736] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2244.280340] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2244.288154] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2244.292212] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2244.295815] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2244.302736] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2244.306335] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2248.057731] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2248.058212] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2248.068063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2248.068184] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2248.068193] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2248.068195] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2248.068216] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2248.068220] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2248.068227] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2248.068229] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2248.068231] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2248.068236] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2248.068241] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2248.068245] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2248.068251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2248.068253] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2248.068260] Call Trace: [ 2248.068269] [ 2248.068276] ? show_regs.cold.9+0x22/0x2f [ 2248.068292] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2248.068302] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2248.068306] ? watchdog+0x40/0x40 [ 2248.068311] ? hrtimer_interrupt+0x128/0x2c0 [ 2248.068318] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2248.068328] ? apic_timer_interrupt+0xf/0x20 [ 2248.068336] [ 2248.068338] ? native_safe_halt+0x2e/0x50 [ 2248.068342] kvm_wait+0xcd/0xe0 [ 2248.068353] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2248.068361] do_raw_spin_lock+0x177/0x190 [ 2248.068367] _raw_spin_lock+0x21/0x30 [ 2248.068371] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2248.082525] Modules linked in: [ 2248.143024] ? do_raw_spin_unlock+0x75/0x190 [ 2248.143046] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2248.143360] ? do_wait_intr+0xf0/0xf0 [ 2248.143368] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2248.153797] lustre(O) osp(O) [ 2248.159092] mdc_close+0x53b/0xd70 [mdc] [ 2248.162768] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2248.162846] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2248.162853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2248.162855] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2248.162868] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2248.162871] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2248.162875] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2248.162877] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2248.162879] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2248.162881] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2248.162952] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2248.162956] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2248.162959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2248.162961] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2248.162969] Call Trace: [ 2248.162974] [ 2248.162979] ? show_regs.cold.9+0x22/0x2f [ 2248.162991] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2248.162997] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2248.163001] ? watchdog+0x40/0x40 [ 2248.163006] ? hrtimer_interrupt+0x128/0x2c0 [ 2248.163010] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2248.163017] ? apic_timer_interrupt+0xf/0x20 [ 2248.163023] [ 2248.163026] ? native_safe_halt+0x2e/0x50 [ 2248.163030] kvm_wait+0xcd/0xe0 [ 2248.163037] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2248.163044] do_raw_spin_lock+0x177/0x190 [ 2248.163048] _raw_spin_lock+0x21/0x30 [ 2248.163052] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2248.163403] obd_mod_free+0x46/0x140 [ptlrpc] [ 2248.173991] lmv_close+0x18d/0x490 [lmv] [ 2248.174034] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2248.178612] mdc_commit_open+0x91/0xb0 [mdc] [ 2248.178681] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2248.182400] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2248.182685] ll_md_close+0x274/0x800 [lustre] [ 2248.182770] ll_file_release+0x128/0xc50 [lustre] [ 2248.182851] __fput+0x10d/0x440 [ 2248.182862] ____fput+0x12/0x20 [ 2248.182866] task_work_run+0xc6/0x110 [ 2248.182873] exit_to_usermode_loop+0x1dd/0x1f0 [ 2248.182881] do_syscall_64+0x3d6/0x440 [ 2248.182958] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2248.182967] RIP: 0033:0x7f1a63983cd8 [ 2248.182975] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2248.182979] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2248.182985] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2248.182987] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2248.182989] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2248.182991] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2248.182993] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2248.717264] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2248.722687] after_reply+0xe2c/0x1780 [ptlrpc] [ 2248.731218] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2248.740385] ? __next_timer_interrupt+0x160/0x160 [ 2248.745826] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2248.750539] ? do_wait_intr+0xf0/0xf0 [ 2248.754140] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2248.760009] kthread+0x1d1/0x200 [ 2248.764782] ? set_kthread_struct+0x70/0x70 [ 2248.769812] ret_from_fork+0x1f/0x30 [ 2262.412056] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 2262.414980] rcu: 3-...!: (137064 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 2262.427295] (detected by 1, t=240006 jiffies, g=269633, q=248) [ 2262.435198] Sending NMI from CPU 1 to CPUs 3: [ 2262.444026] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 3.391 msecs [ 2262.445381] NMI backtrace for cpu 3 [ 2262.445390] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2262.445392] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2262.445393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2262.445395] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2262.445397] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 2262.445400] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2262.445402] RDX: 0000000000005661 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2262.445403] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2262.445404] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2262.445406] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2262.445407] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2262.445408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2262.445410] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2262.445412] Call Trace: [ 2262.445413] [ 2262.445413] ? show_regs.cold.9+0x22/0x2f [ 2262.445414] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2262.445415] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2262.445416] ? nmi_handle+0xbd/0x210 [ 2262.445417] ? default_do_nmi+0x4b/0x1b0 [ 2262.445418] ? do_nmi+0x280/0x380 [ 2262.445419] ? end_repeat_nmi+0x16/0x1f [ 2262.445420] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2262.445421] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2262.445423] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2262.445423] [ 2262.445424] do_raw_spin_lock+0x177/0x190 [ 2262.445425] _raw_spin_lock+0x21/0x30 [ 2262.445427] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2262.445428] obd_mod_free+0x46/0x140 [ptlrpc] [ 2262.445429] mdc_commit_open+0x91/0xb0 [mdc] [ 2262.445430] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2262.445431] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2262.445432] after_reply+0xe2c/0x1780 [ptlrpc] [ 2262.445433] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2262.445434] ? __next_timer_interrupt+0x160/0x160 [ 2262.445435] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2262.445436] ? do_wait_intr+0xf0/0xf0 [ 2262.445437] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2262.445438] kthread+0x1d1/0x200 [ 2262.445439] ? set_kthread_struct+0x70/0x70 [ 2262.445440] ret_from_fork+0x1f/0x30 [ 2262.445445] rcu: rcu_sched kthread starved for 239652 jiffies! g269633 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 2262.716824] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2262.724943] rcu: RCU grace-period kthread stack dump: [ 2262.730825] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2262.742799] Call Trace: [ 2262.748661] __schedule+0x351/0xcb0 [ 2262.758064] schedule+0xc0/0x180 [ 2262.760038] schedule_timeout+0xb4/0x190 [ 2262.767058] ? do_raw_spin_unlock+0x75/0x190 [ 2262.773178] ? __next_timer_interrupt+0x160/0x160 [ 2262.780428] ? prepare_to_swait_event+0xf3/0x140 [ 2262.784625] rcu_gp_kthread+0x8dd/0x15b0 [ 2262.786092] ? force_qs_rnp+0x2c0/0x2c0 [ 2262.788892] kthread+0x1d1/0x200 [ 2262.790672] ? set_kthread_struct+0x70/0x70 [ 2262.801639] ret_from_fork+0x1f/0x30 [ 2262.803079] rcu: Stack dump where RCU GP kthread last ran: [ 2262.809870] Sending NMI from CPU 1 to CPUs 0: [ 2262.818253] NMI backtrace for cpu 0 [ 2262.818260] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2262.818262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2262.818264] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2262.818266] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2262.818268] RSP: 0018:ffffc38541c37500 EFLAGS: 00000216 [ 2262.818272] RAX: 0000000000003690 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2262.818273] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2262.818274] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2262.818276] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2262.818277] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2262.818278] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2262.818280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2262.818281] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2262.818282] Call Trace: [ 2262.818283] [ 2262.818284] ? show_regs.cold.9+0x22/0x2f [ 2262.818285] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2262.818286] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2262.818287] ? nmi_handle+0xbd/0x210 [ 2262.818288] ? default_do_nmi+0x4b/0x1b0 [ 2262.818289] ? do_nmi+0x280/0x380 [ 2262.818289] ? end_repeat_nmi+0x16/0x1f [ 2262.818290] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2262.818292] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2262.818293] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2262.818294] [ 2262.818295] do_raw_spin_lock+0x177/0x190 [ 2262.818296] _raw_spin_lock+0x21/0x30 [ 2262.818296] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2262.818298] ? do_raw_spin_unlock+0x75/0x190 [ 2262.818299] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2262.818300] ? do_wait_intr+0xf0/0xf0 [ 2262.818300] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2262.818302] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2262.818303] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2262.818304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2262.818305] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2262.818306] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2262.818307] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2262.818309] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2262.818310] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2262.818311] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2262.818312] lmv_intent_open+0x338/0x1150 [lmv] [ 2262.818313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2262.818314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2262.818315] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2262.818317] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2262.818318] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2262.818319] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2262.818320] ? do_raw_spin_unlock+0x75/0x190 [ 2262.818321] ? _raw_spin_unlock+0x12/0x30 [ 2262.818323] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2262.818323] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2262.818324] ll_atomic_open+0x36a/0xd70 [lustre] [ 2262.818325] lookup_open+0x356/0xb40 [ 2262.818326] do_last+0x313/0xfc0 [ 2262.818327] ? nd_jump_root+0xe5/0x160 [ 2262.818329] ? path_init+0x437/0x520 [ 2262.818330] path_openat+0xf7/0x500 [ 2262.818331] ? do_raw_spin_unlock+0x75/0x190 [ 2262.818333] do_filp_open+0x99/0x140 [ 2262.818334] ? getname_flags+0x6e/0x330 [ 2262.818335] ? __check_object_size+0xff/0x256 [ 2262.818336] ? do_raw_spin_unlock+0x75/0x190 [ 2262.818337] ? _raw_spin_unlock+0x12/0x30 [ 2262.818338] do_sys_openat2+0x2b4/0x410 [ 2262.818339] do_sys_open+0x73/0xa0 [ 2262.818340] __x64_sys_openat+0x24/0x30 [ 2262.818341] do_syscall_64+0xc1/0x440 [ 2262.818342] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2262.818343] RIP: 0033:0x7f1a63983332 [ 2262.818345] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2262.818347] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2262.818350] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2262.818351] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2262.818353] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2262.818354] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2262.818355] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2270.702458] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 247s! [ 2270.717111] Showing busy workqueues and worker pools: [ 2270.726221] workqueue events: flags=0x0 [ 2270.733941] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2270.741270] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2270.750224] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2270.752694] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2270.770585] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2270.776948] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2270.781097] workqueue events_long: flags=0x0 [ 2270.782452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2270.785655] pending: lu_objects_flush [obdclass] [ 2270.789262] workqueue events_unbound: flags=0x2 [ 2270.791992] pwq 8: cpus=0-3 flags=0x4 nice=0 active=5/512 refcnt=7 [ 2270.793667] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2270.796589] workqueue events_power_efficient: flags=0x80 [ 2270.798854] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2270.800636] pending: neigh_periodic_work, do_cache_clean [ 2270.802417] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2270.806252] pending: neigh_periodic_work, check_lifetime [ 2270.810442] workqueue mm_percpu_wq: flags=0x8 [ 2270.813922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2270.819746] pending: vmstat_update [ 2270.820708] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2270.824273] pending: vmstat_update [ 2270.825498] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2270.827454] pending: vmstat_update [ 2270.828393] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2270.831654] pending: vmstat_update [ 2270.833637] workqueue writeback: flags=0x4a [ 2270.835400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2270.837457] pending: wb_workfn, wb_workfn [ 2270.838948] workqueue xprtiod: flags=0xa [ 2270.841330] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2270.843422] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2270.847067] workqueue ipv6_addrconf: flags=0x40008 [ 2270.848488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2270.851018] pending: addrconf_verify_work [ 2270.852522] workqueue knbd0-recv: flags=0x1a [ 2270.854329] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2270.856529] in-flight: 647:recv_work [ 2270.857945] workqueue ptlrpc_pinger: flags=0x2 [ 2270.859424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2270.861439] pending: ptlrpc_pinger_main [ptlrpc] [ 2270.863370] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2260s workers=2 idle: 652 [ 2276.061779] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2276.064584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2276.103595] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2276.114612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2276.122495] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2276.125599] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2276.136093] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2276.141054] RAX: 0000000000001286 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2276.145240] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2276.148464] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2276.154727] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2276.160453] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2276.167259] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2276.169455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2276.175295] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2276.179365] Call Trace: [ 2276.180482] [ 2276.181166] ? show_regs.cold.9+0x22/0x2f [ 2276.184018] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2276.188281] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2276.191752] ? watchdog+0x40/0x40 [ 2276.194422] ? hrtimer_interrupt+0x128/0x2c0 [ 2276.197248] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2276.199753] ? apic_timer_interrupt+0xf/0x20 [ 2276.201754] [ 2276.202986] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2276.205595] do_raw_spin_lock+0x177/0x190 [ 2276.208070] _raw_spin_lock+0x21/0x30 [ 2276.209873] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2276.213161] ? do_raw_spin_unlock+0x75/0x190 [ 2276.215195] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2276.218434] ? do_wait_intr+0xf0/0xf0 [ 2276.220463] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2276.222698] mdc_close+0x53b/0xd70 [mdc] [ 2276.224984] lmv_close+0x18d/0x490 [lmv] [ 2276.227023] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2276.229969] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2276.232399] ll_md_close+0x274/0x800 [lustre] [ 2276.234613] ll_file_release+0x128/0xc50 [lustre] [ 2276.239312] __fput+0x10d/0x440 [ 2276.240527] ____fput+0x12/0x20 [ 2276.241589] task_work_run+0xc6/0x110 [ 2276.247271] exit_to_usermode_loop+0x1dd/0x1f0 [ 2276.250565] do_syscall_64+0x3d6/0x440 [ 2276.259045] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2276.269098] RIP: 0033:0x7f1a63983cd8 [ 2276.270371] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2276.283922] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2276.287552] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2276.290999] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2276.294040] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2276.300825] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2276.306669] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2288.059056] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2288.060580] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2288.062721] Modules linked in: [ 2288.066797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2288.066893] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2288.066902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2288.066906] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2288.066920] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2288.066924] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2288.066929] RAX: 0000000000003e0f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2288.066931] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2288.066933] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2288.066936] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2288.066938] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2288.066942] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2288.066945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2288.066948] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2288.066956] Call Trace: [ 2288.066962] [ 2288.066968] ? show_regs.cold.9+0x22/0x2f [ 2288.066977] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2288.066983] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2288.066988] ? watchdog+0x40/0x40 [ 2288.066993] ? hrtimer_interrupt+0x128/0x2c0 [ 2288.066996] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2288.067005] ? apic_timer_interrupt+0xf/0x20 [ 2288.067009] [ 2288.067012] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2288.067018] do_raw_spin_lock+0x177/0x190 [ 2288.067023] _raw_spin_lock+0x21/0x30 [ 2288.067029] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2288.067457] ? do_raw_spin_unlock+0x75/0x190 [ 2288.067463] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2288.067727] ? do_wait_intr+0xf0/0xf0 [ 2288.067754] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2288.068108] mdc_close+0x53b/0xd70 [mdc] [ 2288.068145] lmv_close+0x18d/0x490 [lmv] [ 2288.068241] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2288.068401] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2288.068496] ll_md_close+0x274/0x800 [lustre] [ 2288.068646] ll_file_release+0x128/0xc50 [lustre] [ 2288.068764] __fput+0x10d/0x440 [ 2288.068773] ____fput+0x12/0x20 [ 2288.068782] task_work_run+0xc6/0x110 [ 2288.068790] exit_to_usermode_loop+0x1dd/0x1f0 [ 2288.068798] do_syscall_64+0x3d6/0x440 [ 2288.068802] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2288.068809] RIP: 0033:0x7f1a63983cd8 [ 2288.068816] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2288.068820] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2288.068824] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2288.068827] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2288.068829] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2288.068831] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2288.068834] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2288.230796] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2288.252767] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2288.257372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2288.260849] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2288.262813] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2288.273024] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2288.278298] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2288.280925] RDX: 0000000000005d35 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2288.284197] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2288.287617] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2288.289945] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2288.293433] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2288.297349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2288.300391] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2288.305133] Call Trace: [ 2288.306926] [ 2288.308008] ? show_regs.cold.9+0x22/0x2f [ 2288.309906] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2288.312324] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2288.314710] ? watchdog+0x40/0x40 [ 2288.316525] ? hrtimer_interrupt+0x128/0x2c0 [ 2288.318231] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2288.320386] ? apic_timer_interrupt+0xf/0x20 [ 2288.322403] [ 2288.323483] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2288.329296] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2288.331139] do_raw_spin_lock+0x177/0x190 [ 2288.333613] _raw_spin_lock+0x21/0x30 [ 2288.334874] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2288.337688] obd_mod_free+0x46/0x140 [ptlrpc] [ 2288.339806] mdc_commit_open+0x91/0xb0 [mdc] [ 2288.341208] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2288.343668] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2288.347093] after_reply+0xe2c/0x1780 [ptlrpc] [ 2288.350557] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2288.354238] ? __next_timer_interrupt+0x160/0x160 [ 2288.355986] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2288.360539] ? do_wait_intr+0xf0/0xf0 [ 2288.363617] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2288.366934] kthread+0x1d1/0x200 [ 2288.368543] ? set_kthread_struct+0x70/0x70 [ 2288.372204] ret_from_fork+0x1f/0x30 [ 2296.053245] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2296.064140] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2296.122240] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2296.137516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2296.148077] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2296.150892] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2296.166436] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2296.172290] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2296.176596] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2296.180165] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2296.184128] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2296.190992] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2296.199436] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2296.212549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2296.222437] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2296.232987] Call Trace: [ 2296.236412] [ 2296.238162] ? show_regs.cold.9+0x22/0x2f [ 2296.245964] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2296.250190] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2296.253141] ? watchdog+0x40/0x40 [ 2296.255211] ? hrtimer_interrupt+0x128/0x2c0 [ 2296.257205] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2296.258875] ? apic_timer_interrupt+0xf/0x20 [ 2296.261311] [ 2296.262433] ? native_safe_halt+0x2e/0x50 [ 2296.264349] kvm_wait+0xcd/0xe0 [ 2296.266245] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2296.270812] do_raw_spin_lock+0x177/0x190 [ 2296.275229] _raw_spin_lock+0x21/0x30 [ 2296.277417] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2296.281210] ? do_raw_spin_unlock+0x75/0x190 [ 2296.283288] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2296.288162] ? do_wait_intr+0xf0/0xf0 [ 2296.291326] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2296.298172] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2296.304500] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2296.314903] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2296.318920] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2296.322146] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2296.325531] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2296.327907] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2296.330529] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2296.333817] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2296.336341] lmv_intent_open+0x338/0x1150 [lmv] [ 2296.341275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2296.343683] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2296.350521] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2296.355584] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2296.361245] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2296.364177] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2296.366550] ? do_raw_spin_unlock+0x75/0x190 [ 2296.370227] ? _raw_spin_unlock+0x12/0x30 [ 2296.375259] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2296.380512] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2296.385884] ll_atomic_open+0x36a/0xd70 [lustre] [ 2296.389940] lookup_open+0x356/0xb40 [ 2296.395212] do_last+0x313/0xfc0 [ 2296.398566] ? nd_jump_root+0xe5/0x160 [ 2296.400779] ? path_init+0x437/0x520 [ 2296.402464] path_openat+0xf7/0x500 [ 2296.404139] ? do_raw_spin_unlock+0x75/0x190 [ 2296.409301] do_filp_open+0x99/0x140 [ 2296.411529] ? getname_flags+0x6e/0x330 [ 2296.413567] ? __check_object_size+0xff/0x256 [ 2296.416296] ? do_raw_spin_unlock+0x75/0x190 [ 2296.418919] ? _raw_spin_unlock+0x12/0x30 [ 2296.420683] do_sys_openat2+0x2b4/0x410 [ 2296.424968] do_sys_open+0x73/0xa0 [ 2296.426642] __x64_sys_openat+0x24/0x30 [ 2296.428631] do_syscall_64+0xc1/0x440 [ 2296.430186] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2296.432841] RIP: 0033:0x7f1a63983332 [ 2296.434939] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2296.443701] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2296.446841] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2296.449492] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2296.453533] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2296.457286] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2296.463339] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2300.896047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 277s! [ 2300.901047] Showing busy workqueues and worker pools: [ 2300.907583] workqueue events: flags=0x0 [ 2300.909588] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2300.912602] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2300.916331] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2300.920319] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2300.935382] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2300.940836] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2300.952859] workqueue events_long: flags=0x0 [ 2300.960416] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2300.968042] pending: lu_objects_flush [obdclass] [ 2300.974069] workqueue events_unbound: flags=0x2 [ 2300.981705] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2300.984415] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2300.990826] workqueue events_power_efficient: flags=0x80 [ 2300.993240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2300.997845] pending: neigh_periodic_work, do_cache_clean [ 2301.000853] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2301.004481] pending: neigh_periodic_work, check_lifetime [ 2301.007586] workqueue mm_percpu_wq: flags=0x8 [ 2301.010042] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2301.013550] pending: vmstat_update [ 2301.016831] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2301.025862] pending: vmstat_update [ 2301.029788] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2301.038040] pending: vmstat_update [ 2301.040958] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2301.049821] pending: vmstat_update [ 2301.052926] workqueue writeback: flags=0x4a [ 2301.054269] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2301.064512] pending: wb_workfn, wb_workfn [ 2301.067049] workqueue xprtiod: flags=0xa [ 2301.069153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2301.073044] pending: xs_stream_data_receive_workfn, xs_error_handle [ 2301.078535] workqueue ipv6_addrconf: flags=0x40008 [ 2301.080988] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2301.084677] pending: addrconf_verify_work [ 2301.088918] workqueue knbd0-recv: flags=0x1a [ 2301.093860] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2301.101874] in-flight: 647:recv_work [ 2301.107091] workqueue ptlrpc_pinger: flags=0x2 [ 2301.115789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2301.120059] pending: ptlrpc_pinger_main [ptlrpc] [ 2301.127524] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2291s workers=2 idle: 652 [ 2304.057031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2304.061081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2304.082994] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2304.087768] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2304.091509] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2304.093493] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2304.101447] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2304.104392] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2304.108201] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2304.111771] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2304.114424] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2304.117161] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2304.120008] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2304.123642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2304.125954] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2304.128739] Call Trace: [ 2304.129918] [ 2304.130778] ? show_regs.cold.9+0x22/0x2f [ 2304.132981] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2304.136461] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2304.141819] ? watchdog+0x40/0x40 [ 2304.143511] ? hrtimer_interrupt+0x128/0x2c0 [ 2304.146889] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2304.149069] ? apic_timer_interrupt+0xf/0x20 [ 2304.150716] [ 2304.151698] ? native_safe_halt+0x2e/0x50 [ 2304.153686] kvm_wait+0xcd/0xe0 [ 2304.155162] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2304.156818] do_raw_spin_lock+0x177/0x190 [ 2304.158673] _raw_spin_lock+0x21/0x30 [ 2304.160036] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2304.162502] ? do_raw_spin_unlock+0x75/0x190 [ 2304.164284] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2304.166051] ? do_wait_intr+0xf0/0xf0 [ 2304.171265] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2304.173633] mdc_close+0x53b/0xd70 [mdc] [ 2304.176857] lmv_close+0x18d/0x490 [lmv] [ 2304.178871] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2304.182812] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2304.189252] ll_md_close+0x274/0x800 [lustre] [ 2304.192021] ll_file_release+0x128/0xc50 [lustre] [ 2304.193989] __fput+0x10d/0x440 [ 2304.195139] ____fput+0x12/0x20 [ 2304.197564] task_work_run+0xc6/0x110 [ 2304.199065] exit_to_usermode_loop+0x1dd/0x1f0 [ 2304.208562] do_syscall_64+0x3d6/0x440 [ 2304.209941] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2304.214555] RIP: 0033:0x7f1a63983cd8 [ 2304.216277] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2304.226847] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2304.230336] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2304.238549] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2304.244543] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2304.248201] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2304.254562] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2316.056643] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2316.058487] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2316.062030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [ 2316.073746] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2316.073829] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2316.073836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2316.073838] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2316.073852] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2316.073856] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2316.073860] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2316.073863] RDX: 0000000000000276 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2316.073865] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2316.073867] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2316.073869] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2316.073873] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2316.073875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2316.073877] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2316.073887] Call Trace: [ 2316.073892] [ 2316.073897] ? show_regs.cold.9+0x22/0x2f [ 2316.073906] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2316.073912] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2316.073916] ? watchdog+0x40/0x40 [ 2316.073921] ? hrtimer_interrupt+0x128/0x2c0 [ 2316.073925] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2316.073932] ? apic_timer_interrupt+0xf/0x20 [ 2316.073937] [ 2316.073940] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2316.073944] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2316.073948] do_raw_spin_lock+0x177/0x190 [ 2316.073953] _raw_spin_lock+0x21/0x30 [ 2316.073958] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2316.074630] obd_mod_free+0x46/0x140 [ptlrpc] [ 2316.095046] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2316.095094] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2316.095104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2316.095108] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2316.095125] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2316.095130] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2316.095135] RAX: 0000000000007f83 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2316.095137] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2316.095140] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2316.095142] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2316.095145] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2316.095148] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2316.095151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2316.095154] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2316.095163] Call Trace: [ 2316.095169] [ 2316.095176] ? show_regs.cold.9+0x22/0x2f [ 2316.095187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2316.095193] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2316.095198] ? watchdog+0x40/0x40 [ 2316.095202] ? hrtimer_interrupt+0x128/0x2c0 [ 2316.095206] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2316.095215] ? apic_timer_interrupt+0xf/0x20 [ 2316.095219] [ 2316.095222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2316.095228] do_raw_spin_lock+0x177/0x190 [ 2316.095233] _raw_spin_lock+0x21/0x30 [ 2316.095238] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2316.134678] mdc_commit_open+0x91/0xb0 [mdc] [ 2316.138779] ? do_raw_spin_unlock+0x75/0x190 [ 2316.138799] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2316.144903] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2316.147111] ? do_wait_intr+0xf0/0xf0 [ 2316.147130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2316.147416] mdc_close+0x53b/0xd70 [mdc] [ 2316.160445] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2316.169975] lmv_close+0x18d/0x490 [lmv] [ 2316.174620] after_reply+0xe2c/0x1780 [ptlrpc] [ 2316.175173] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2316.175509] ? __next_timer_interrupt+0x160/0x160 [ 2316.175520] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2316.175941] ? do_wait_intr+0xf0/0xf0 [ 2316.175950] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2316.176382] kthread+0x1d1/0x200 [ 2316.176387] ? set_kthread_struct+0x70/0x70 [ 2316.176392] ret_from_fork+0x1f/0x30 [ 2316.471401] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2316.473534] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2316.482201] ll_md_close+0x274/0x800 [lustre] [ 2316.487449] ll_file_release+0x128/0xc50 [lustre] [ 2316.495075] __fput+0x10d/0x440 [ 2316.498414] ____fput+0x12/0x20 [ 2316.501380] task_work_run+0xc6/0x110 [ 2316.502744] exit_to_usermode_loop+0x1dd/0x1f0 [ 2316.505634] do_syscall_64+0x3d6/0x440 [ 2316.507253] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2316.513161] RIP: 0033:0x7f1a63983cd8 [ 2316.522190] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2316.543320] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2316.550262] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2316.565860] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2316.574556] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2316.589234] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2316.595165] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2328.059503] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 2328.081586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2328.144933] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2328.159483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2328.172518] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2328.176849] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2328.196811] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2328.210839] RAX: 0000000000005208 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2328.216190] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2328.226181] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2328.234998] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2328.241306] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2328.248309] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2328.252735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2328.266699] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2328.278261] Call Trace: [ 2328.279150] [ 2328.280275] ? show_regs.cold.9+0x22/0x2f [ 2328.283907] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2328.286902] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2328.289850] ? watchdog+0x40/0x40 [ 2328.292112] ? hrtimer_interrupt+0x128/0x2c0 [ 2328.296406] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2328.300824] ? apic_timer_interrupt+0xf/0x20 [ 2328.303419] [ 2328.304666] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2328.310000] do_raw_spin_lock+0x177/0x190 [ 2328.312742] _raw_spin_lock+0x21/0x30 [ 2328.315249] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2328.318888] ? do_raw_spin_unlock+0x75/0x190 [ 2328.320726] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2328.326693] ? do_wait_intr+0xf0/0xf0 [ 2328.329644] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2328.334250] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2328.341743] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2328.348032] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2328.352408] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2328.356542] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2328.360575] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2328.363060] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2328.367607] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2328.373619] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2328.376029] lmv_intent_open+0x338/0x1150 [lmv] [ 2328.380048] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2328.382270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2328.385130] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2328.387432] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2328.390152] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2328.394709] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2328.397509] ? do_raw_spin_unlock+0x75/0x190 [ 2328.402918] ? _raw_spin_unlock+0x12/0x30 [ 2328.408714] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2328.411087] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2328.416150] ll_atomic_open+0x36a/0xd70 [lustre] [ 2328.420796] lookup_open+0x356/0xb40 [ 2328.423519] do_last+0x313/0xfc0 [ 2328.425360] ? nd_jump_root+0xe5/0x160 [ 2328.430634] ? path_init+0x437/0x520 [ 2328.435829] path_openat+0xf7/0x500 [ 2328.439620] ? do_raw_spin_unlock+0x75/0x190 [ 2328.444446] do_filp_open+0x99/0x140 [ 2328.449710] ? getname_flags+0x6e/0x330 [ 2328.452549] ? __check_object_size+0xff/0x256 [ 2328.454325] ? do_raw_spin_unlock+0x75/0x190 [ 2328.456855] ? _raw_spin_unlock+0x12/0x30 [ 2328.458339] do_sys_openat2+0x2b4/0x410 [ 2328.460535] do_sys_open+0x73/0xa0 [ 2328.462287] __x64_sys_openat+0x24/0x30 [ 2328.464168] do_syscall_64+0xc1/0x440 [ 2328.469690] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2328.474623] RIP: 0033:0x7f1a63983332 [ 2328.480343] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2328.491861] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2328.495957] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2328.500601] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2328.504168] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2328.507309] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2328.513955] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2331.619018] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 308s! [ 2331.628726] Showing busy workqueues and worker pools: [ 2331.635223] workqueue events: flags=0x0 [ 2331.642293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2331.653565] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2331.669435] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2331.676639] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2331.687529] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2331.691831] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2331.702883] workqueue events_long: flags=0x0 [ 2331.705516] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2331.716957] pending: lu_objects_flush [obdclass] [ 2331.723644] workqueue events_unbound: flags=0x2 [ 2331.730494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2331.734953] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2331.740557] workqueue events_power_efficient: flags=0x80 [ 2331.749067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2331.753295] pending: neigh_periodic_work, do_cache_clean [ 2331.761659] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2331.771612] pending: neigh_periodic_work, check_lifetime [ 2331.778016] workqueue mm_percpu_wq: flags=0x8 [ 2331.782937] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2331.786871] pending: vmstat_update [ 2331.803723] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2331.820040] pending: vmstat_update [ 2331.822784] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2331.828952] pending: vmstat_update [ 2331.830229] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2331.841525] pending: vmstat_update [ 2331.844591] workqueue writeback: flags=0x4a [ 2331.849827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2331.859082] pending: wb_workfn, wb_workfn [ 2331.863806] workqueue xprtiod: flags=0xa [ 2331.870931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2331.879975] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2331.888935] workqueue ipv6_addrconf: flags=0x40008 [ 2331.895025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2331.906663] pending: addrconf_verify_work [ 2331.911354] workqueue knbd0-recv: flags=0x1a [ 2331.912852] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2331.921759] in-flight: 647:recv_work [ 2331.925124] workqueue ptlrpc_pinger: flags=0x2 [ 2331.928526] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2331.936877] pending: ptlrpc_pinger_main [ptlrpc] [ 2331.943371] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2321s workers=2 idle: 652 [ 2332.057080] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2332.060175] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2332.082391] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2332.088034] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2332.092191] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2332.094940] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2332.103925] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2332.107934] RAX: 0000000000007b4a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2332.111233] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2332.114924] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2332.118230] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2332.121830] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2332.125388] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2332.129424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2332.131982] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2332.135773] Call Trace: [ 2332.137011] [ 2332.137921] ? show_regs.cold.9+0x22/0x2f [ 2332.140690] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2332.143147] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2332.145492] ? watchdog+0x40/0x40 [ 2332.146967] ? hrtimer_interrupt+0x128/0x2c0 [ 2332.151680] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2332.154032] ? apic_timer_interrupt+0xf/0x20 [ 2332.156645] [ 2332.157549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2332.160003] do_raw_spin_lock+0x177/0x190 [ 2332.162407] _raw_spin_lock+0x21/0x30 [ 2332.164387] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2332.167639] ? do_raw_spin_unlock+0x75/0x190 [ 2332.169945] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2332.172991] ? do_wait_intr+0xf0/0xf0 [ 2332.174951] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2332.178522] mdc_close+0x53b/0xd70 [mdc] [ 2332.180920] lmv_close+0x18d/0x490 [lmv] [ 2332.183740] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2332.186745] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2332.189029] ll_md_close+0x274/0x800 [lustre] [ 2332.191692] ll_file_release+0x128/0xc50 [lustre] [ 2332.195174] __fput+0x10d/0x440 [ 2332.197027] ____fput+0x12/0x20 [ 2332.199620] task_work_run+0xc6/0x110 [ 2332.202277] exit_to_usermode_loop+0x1dd/0x1f0 [ 2332.204555] do_syscall_64+0x3d6/0x440 [ 2332.206779] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2332.209903] RIP: 0033:0x7f1a63983cd8 [ 2332.211799] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2332.220885] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2332.224297] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2332.227891] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2332.231764] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2332.235112] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2332.238766] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2344.056018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 2344.058019] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 2344.063865] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 2344.068998] Modules linked in: [ 2344.068999] intel_rapl_common sb_edac rapl [ 2344.085339] lustre(O) [ 2344.086719] pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2344.086747] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2344.086752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2344.086755] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2344.086767] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2344.086770] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2344.086774] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2344.086776] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2344.086778] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2344.086781] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2344.086783] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2344.086785] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2344.086788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2344.086790] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2344.086797] Call Trace: [ 2344.086801] [ 2344.086806] ? show_regs.cold.9+0x22/0x2f [ 2344.086815] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2344.086821] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2344.086824] ? watchdog+0x40/0x40 [ 2344.086828] ? hrtimer_interrupt+0x128/0x2c0 [ 2344.086831] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2344.086837] ? apic_timer_interrupt+0xf/0x20 [ 2344.086841] [ 2344.086844] ? native_safe_halt+0x2e/0x50 [ 2344.086846] kvm_wait+0xcd/0xe0 [ 2344.086853] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2344.086861] do_raw_spin_lock+0x177/0x190 [ 2344.086864] _raw_spin_lock+0x21/0x30 [ 2344.086867] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2344.088594] osp(O) [ 2344.089170] ? do_raw_spin_unlock+0x75/0x190 [ 2344.099815] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2344.099891] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2344.099900] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2344.099903] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2344.099917] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2344.099921] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2344.100152] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2344.100155] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2344.100157] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2344.100159] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2344.100162] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2344.100164] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2344.100167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2344.100169] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2344.100177] Call Trace: [ 2344.100183] [ 2344.100188] ? show_regs.cold.9+0x22/0x2f [ 2344.100213] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2344.100220] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2344.100224] ? watchdog+0x40/0x40 [ 2344.100228] ? hrtimer_interrupt+0x128/0x2c0 [ 2344.100233] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2344.100239] ? apic_timer_interrupt+0xf/0x20 [ 2344.100244] [ 2344.100247] ? native_safe_halt+0x2e/0x50 [ 2344.100250] kvm_wait+0xcd/0xe0 [ 2344.100259] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2344.100265] do_raw_spin_lock+0x177/0x190 [ 2344.100269] _raw_spin_lock+0x21/0x30 [ 2344.100274] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2344.100570] obd_mod_free+0x46/0x140 [ptlrpc] [ 2344.101076] mdc_commit_open+0x91/0xb0 [mdc] [ 2344.101111] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2344.101323] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2344.106947] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2344.113160] after_reply+0xe2c/0x1780 [ptlrpc] [ 2344.118538] ? do_wait_intr+0xf0/0xf0 [ 2344.127543] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2344.127865] ? __next_timer_interrupt+0x160/0x160 [ 2344.127876] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2344.133552] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2344.136877] ? do_wait_intr+0xf0/0xf0 [ 2344.136902] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2344.140763] mdc_close+0x53b/0xd70 [mdc] [ 2344.143826] kthread+0x1d1/0x200 [ 2344.149643] lmv_close+0x18d/0x490 [lmv] [ 2344.156289] ? set_kthread_struct+0x70/0x70 [ 2344.156310] ret_from_fork+0x1f/0x30 [ 2344.384763] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2344.387492] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2344.390564] ll_md_close+0x274/0x800 [lustre] [ 2344.397100] ll_file_release+0x128/0xc50 [lustre] [ 2344.402965] __fput+0x10d/0x440 [ 2344.404764] ____fput+0x12/0x20 [ 2344.406767] task_work_run+0xc6/0x110 [ 2344.408294] exit_to_usermode_loop+0x1dd/0x1f0 [ 2344.412959] do_syscall_64+0x3d6/0x440 [ 2344.414733] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2344.418454] RIP: 0033:0x7f1a63983cd8 [ 2344.420563] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2344.438585] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2344.442256] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2344.445864] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2344.455541] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2344.465856] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2344.469042] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2356.056271] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 2356.062596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2356.104594] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2356.112805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2356.125566] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2356.129733] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2356.145093] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2356.148154] RAX: 000000000000136a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2356.151528] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2356.160157] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2356.165740] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2356.171670] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2356.175172] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2356.179498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2356.183277] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2356.187918] Call Trace: [ 2356.189366] [ 2356.190655] ? show_regs.cold.9+0x22/0x2f [ 2356.193053] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2356.195629] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2356.198608] ? watchdog+0x40/0x40 [ 2356.200278] ? hrtimer_interrupt+0x128/0x2c0 [ 2356.202562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2356.204980] ? apic_timer_interrupt+0xf/0x20 [ 2356.207007] [ 2356.208061] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2356.212614] do_raw_spin_lock+0x177/0x190 [ 2356.215655] _raw_spin_lock+0x21/0x30 [ 2356.219698] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2356.225880] ? do_raw_spin_unlock+0x75/0x190 [ 2356.228828] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2356.234521] ? do_wait_intr+0xf0/0xf0 [ 2356.237982] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2356.242278] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2356.245126] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2356.250472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2356.255790] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2356.262933] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2356.270753] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2356.273420] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2356.276908] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2356.281973] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2356.284858] lmv_intent_open+0x338/0x1150 [lmv] [ 2356.287136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2356.293622] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2356.302790] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2356.306767] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2356.309451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2356.311886] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2356.314066] ? do_raw_spin_unlock+0x75/0x190 [ 2356.316184] ? _raw_spin_unlock+0x12/0x30 [ 2356.317950] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2356.320317] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2356.322642] ll_atomic_open+0x36a/0xd70 [lustre] [ 2356.324648] lookup_open+0x356/0xb40 [ 2356.333370] do_last+0x313/0xfc0 [ 2356.336546] ? nd_jump_root+0xe5/0x160 [ 2356.341617] ? path_init+0x437/0x520 [ 2356.345543] path_openat+0xf7/0x500 [ 2356.347806] ? do_raw_spin_unlock+0x75/0x190 [ 2356.349491] do_filp_open+0x99/0x140 [ 2356.350896] ? getname_flags+0x6e/0x330 [ 2356.353691] ? __check_object_size+0xff/0x256 [ 2356.356752] ? do_raw_spin_unlock+0x75/0x190 [ 2356.359288] ? _raw_spin_unlock+0x12/0x30 [ 2356.361665] do_sys_openat2+0x2b4/0x410 [ 2356.363465] do_sys_open+0x73/0xa0 [ 2356.364800] __x64_sys_openat+0x24/0x30 [ 2356.370529] do_syscall_64+0xc1/0x440 [ 2356.374400] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2356.378807] RIP: 0033:0x7f1a63983332 [ 2356.381816] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2356.399295] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2356.406498] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2356.414240] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2356.419661] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2356.423670] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2356.444335] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2360.057594] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2360.063674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2360.090800] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2360.099876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2360.104468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2360.107746] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2360.118299] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2360.122474] RAX: 0000000000007e1d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2360.126167] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2360.130358] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2360.134000] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2360.138172] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2360.142018] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2360.146681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2360.150570] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2360.153889] Call Trace: [ 2360.154924] [ 2360.155819] ? show_regs.cold.9+0x22/0x2f [ 2360.157525] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2360.159572] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2360.161184] ? watchdog+0x40/0x40 [ 2360.162346] ? hrtimer_interrupt+0x128/0x2c0 [ 2360.163657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2360.165385] ? apic_timer_interrupt+0xf/0x20 [ 2360.166806] [ 2360.167701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2360.170170] do_raw_spin_lock+0x177/0x190 [ 2360.171922] _raw_spin_lock+0x21/0x30 [ 2360.173695] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2360.176189] ? do_raw_spin_unlock+0x75/0x190 [ 2360.178193] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2360.181107] ? do_wait_intr+0xf0/0xf0 [ 2360.182691] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2360.185246] mdc_close+0x53b/0xd70 [mdc] [ 2360.186949] lmv_close+0x18d/0x490 [lmv] [ 2360.189122] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2360.191590] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2360.193603] ll_md_close+0x274/0x800 [lustre] [ 2360.195716] ll_file_release+0x128/0xc50 [lustre] [ 2360.198105] __fput+0x10d/0x440 [ 2360.199367] ____fput+0x12/0x20 [ 2360.200516] task_work_run+0xc6/0x110 [ 2360.202519] exit_to_usermode_loop+0x1dd/0x1f0 [ 2360.204363] do_syscall_64+0x3d6/0x440 [ 2360.205871] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2360.208351] RIP: 0033:0x7f1a63983cd8 [ 2360.210002] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2360.218292] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2360.221260] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2360.224291] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2360.227248] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2360.230444] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2360.233682] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2362.342301] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 339s! [ 2362.352273] Showing busy workqueues and worker pools: [ 2362.354603] workqueue events: flags=0x0 [ 2362.356011] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2362.361398] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2362.369444] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2362.379122] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2362.384291] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2362.387298] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2362.391239] workqueue events_long: flags=0x0 [ 2362.392873] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2362.394976] pending: lu_objects_flush [obdclass] [ 2362.396882] workqueue events_unbound: flags=0x2 [ 2362.398594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2362.400768] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2362.404876] workqueue events_power_efficient: flags=0x80 [ 2362.407099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2362.409954] pending: neigh_periodic_work, do_cache_clean [ 2362.412200] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2362.414493] pending: neigh_periodic_work, check_lifetime [ 2362.416568] workqueue mm_percpu_wq: flags=0x8 [ 2362.424192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2362.427391] pending: vmstat_update [ 2362.432976] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2362.441366] pending: vmstat_update [ 2362.449505] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2362.455711] pending: vmstat_update [ 2362.458464] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2362.461963] pending: vmstat_update [ 2362.463510] workqueue writeback: flags=0x4a [ 2362.465471] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2362.468126] pending: wb_workfn, wb_workfn [ 2362.471964] workqueue xprtiod: flags=0xa [ 2362.475566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2362.478508] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2362.482146] workqueue ipv6_addrconf: flags=0x40008 [ 2362.483580] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2362.487388] pending: addrconf_verify_work [ 2362.491392] workqueue knbd0-recv: flags=0x1a [ 2362.496224] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2362.504438] in-flight: 647:recv_work [ 2362.507478] workqueue ptlrpc_pinger: flags=0x2 [ 2362.512493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2362.516949] pending: ptlrpc_pinger_main [ptlrpc] [ 2362.521675] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2352s workers=2 idle: 652 [ 2372.056042] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 2372.061589] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2372.065026] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 2372.138806] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2372.138820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2372.138822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2372.138836] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2372.138840] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2372.138846] RAX: 000000000000668b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2372.138848] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2372.138849] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2372.138851] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2372.138854] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2372.138857] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2372.138859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2372.138861] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2372.138870] Call Trace: [ 2372.138876] [ 2372.138881] ? show_regs.cold.9+0x22/0x2f [ 2372.138891] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2372.138897] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2372.138901] ? watchdog+0x40/0x40 [ 2372.138906] ? hrtimer_interrupt+0x128/0x2c0 [ 2372.138909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2372.138917] ? apic_timer_interrupt+0xf/0x20 [ 2372.138921] [ 2372.138932] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2372.138941] do_raw_spin_lock+0x177/0x190 [ 2372.138946] _raw_spin_lock+0x21/0x30 [ 2372.138951] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2372.158780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2372.158990] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2372.158997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2372.159000] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2372.159014] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2372.159018] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2372.159022] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2372.159025] RDX: 00000000000044c9 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2372.159027] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2372.159029] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2372.159032] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2372.159035] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2372.159037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2372.159039] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2372.159047] Call Trace: [ 2372.159051] [ 2372.159057] ? show_regs.cold.9+0x22/0x2f [ 2372.159078] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2372.159085] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2372.159088] ? watchdog+0x40/0x40 [ 2372.159092] ? hrtimer_interrupt+0x128/0x2c0 [ 2372.159096] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2372.159109] ? apic_timer_interrupt+0xf/0x20 [ 2372.159114] [ 2372.159118] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2372.159122] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2372.159127] do_raw_spin_lock+0x177/0x190 [ 2372.159131] _raw_spin_lock+0x21/0x30 [ 2372.159136] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2372.179589] ? do_raw_spin_unlock+0x75/0x190 [ 2372.180275] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2372.182136] ? do_wait_intr+0xf0/0xf0 [ 2372.190416] obd_mod_free+0x46/0x140 [ptlrpc] [ 2372.194268] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2372.194774] mdc_close+0x53b/0xd70 [mdc] [ 2372.194814] lmv_close+0x18d/0x490 [lmv] [ 2372.194842] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2372.194979] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2372.195294] ll_md_close+0x274/0x800 [lustre] [ 2372.195380] ll_file_release+0x128/0xc50 [lustre] [ 2372.195545] __fput+0x10d/0x440 [ 2372.195556] ____fput+0x12/0x20 [ 2372.195560] task_work_run+0xc6/0x110 [ 2372.195569] exit_to_usermode_loop+0x1dd/0x1f0 [ 2372.195577] do_syscall_64+0x3d6/0x440 [ 2372.195581] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2372.195592] RIP: 0033:0x7f1a63983cd8 [ 2372.195599] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2372.195602] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2372.195607] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2372.195609] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2372.195611] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2372.195613] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2372.195615] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2372.734750] mdc_commit_open+0x91/0xb0 [mdc] [ 2372.740030] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2372.742034] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2372.744503] after_reply+0xe2c/0x1780 [ptlrpc] [ 2372.746521] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2372.748532] ? __next_timer_interrupt+0x160/0x160 [ 2372.750352] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2372.752408] ? do_wait_intr+0xf0/0xf0 [ 2372.754036] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2372.756520] kthread+0x1d1/0x200 [ 2372.758014] ? set_kthread_struct+0x70/0x70 [ 2372.760061] ret_from_fork+0x1f/0x30 [ 2388.053241] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 2388.058261] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2388.058652] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 2388.085997] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2388.086010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2388.086013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2388.086027] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2388.086031] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2388.086036] RAX: 0000000000005b26 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2388.086038] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2388.086040] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2388.086042] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2388.086044] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2388.086047] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2388.086050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2388.086052] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2388.086060] Call Trace: [ 2388.086077] [ 2388.086082] ? show_regs.cold.9+0x22/0x2f [ 2388.086091] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2388.086097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2388.086102] ? watchdog+0x40/0x40 [ 2388.086105] ? hrtimer_interrupt+0x128/0x2c0 [ 2388.086109] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2388.086118] ? apic_timer_interrupt+0xf/0x20 [ 2388.086122] [ 2388.086124] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2388.086342] do_raw_spin_lock+0x177/0x190 [ 2388.086348] _raw_spin_lock+0x21/0x30 [ 2388.086353] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2388.086689] ? do_raw_spin_unlock+0x75/0x190 [ 2388.086695] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2388.095347] Modules linked in: lustre(O) [ 2388.100570] ? do_wait_intr+0xf0/0xf0 [ 2388.100593] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2388.100899] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2388.101243] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2388.101454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2388.101621] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2388.101661] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2388.101691] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2388.101720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2388.101816] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2388.106526] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 2388.109047] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2388.109234] lmv_intent_open+0x338/0x1150 [lmv] [ 2388.109270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2388.118620] lnet(O) [ 2388.125045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2388.128462] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2388.128510] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2388.128518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2388.128521] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2388.131673] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2388.131725] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2388.132025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2388.132170] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2388.132252] ? do_raw_spin_unlock+0x75/0x190 [ 2388.132261] ? _raw_spin_unlock+0x12/0x30 [ 2388.132268] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2388.132301] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2388.132318] ll_atomic_open+0x36a/0xd70 [lustre] [ 2388.132402] lookup_open+0x356/0xb40 [ 2388.132411] do_last+0x313/0xfc0 [ 2388.132416] ? nd_jump_root+0xe5/0x160 [ 2388.132420] ? path_init+0x437/0x520 [ 2388.132424] path_openat+0xf7/0x500 [ 2388.132429] ? do_raw_spin_unlock+0x75/0x190 [ 2388.132433] do_filp_open+0x99/0x140 [ 2388.132440] ? getname_flags+0x6e/0x330 [ 2388.132444] ? __check_object_size+0xff/0x256 [ 2388.132450] ? do_raw_spin_unlock+0x75/0x190 [ 2388.132454] ? _raw_spin_unlock+0x12/0x30 [ 2388.132459] do_sys_openat2+0x2b4/0x410 [ 2388.132463] do_sys_open+0x73/0xa0 [ 2388.132468] __x64_sys_openat+0x24/0x30 [ 2388.132471] do_syscall_64+0xc1/0x440 [ 2388.132478] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2388.132484] RIP: 0033:0x7f1a63983332 [ 2388.132493] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2388.132497] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2388.132503] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2388.132505] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2388.132508] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2388.132510] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2388.132512] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2388.493943] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2388.505118] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2388.509085] RAX: 00000000000066b4 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2388.515733] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2388.523232] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2388.540100] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2388.545735] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2388.552025] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2388.562886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2388.569329] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2388.579111] Call Trace: [ 2388.584190] [ 2388.585000] ? show_regs.cold.9+0x22/0x2f [ 2388.588940] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2388.591011] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2388.603430] ? watchdog+0x40/0x40 [ 2388.605520] ? hrtimer_interrupt+0x128/0x2c0 [ 2388.609166] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2388.618956] ? apic_timer_interrupt+0xf/0x20 [ 2388.633438] [ 2388.634322] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2388.636237] do_raw_spin_lock+0x177/0x190 [ 2388.645132] _raw_spin_lock+0x21/0x30 [ 2388.646432] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2388.652018] ? do_raw_spin_unlock+0x75/0x190 [ 2388.653576] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2388.660855] ? do_wait_intr+0xf0/0xf0 [ 2388.662213] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2388.667931] mdc_close+0x53b/0xd70 [mdc] [ 2388.669459] lmv_close+0x18d/0x490 [lmv] [ 2388.673602] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2388.679412] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2388.681635] ll_md_close+0x274/0x800 [lustre] [ 2388.683745] ll_file_release+0x128/0xc50 [lustre] [ 2388.685998] __fput+0x10d/0x440 [ 2388.687945] ____fput+0x12/0x20 [ 2388.689172] task_work_run+0xc6/0x110 [ 2388.690570] exit_to_usermode_loop+0x1dd/0x1f0 [ 2388.692551] do_syscall_64+0x3d6/0x440 [ 2388.698701] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2388.700982] RIP: 0033:0x7f1a63983cd8 [ 2388.702880] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2388.716753] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2388.722466] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2388.725549] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2388.734290] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2388.737719] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2388.740787] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2392.550816] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 369s! [ 2392.566082] Showing busy workqueues and worker pools: [ 2392.570033] workqueue events: flags=0x0 [ 2392.579213] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2392.588424] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2392.595701] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2392.604206] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2392.612294] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2392.620399] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2392.633379] workqueue events_long: flags=0x0 [ 2392.641312] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2392.650483] pending: lu_objects_flush [obdclass] [ 2392.656141] workqueue events_unbound: flags=0x2 [ 2392.668610] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2392.679703] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2392.696364] workqueue events_power_efficient: flags=0x80 [ 2392.702984] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2392.712713] pending: neigh_periodic_work, do_cache_clean [ 2392.723146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2392.737270] pending: neigh_periodic_work, check_lifetime [ 2392.742704] workqueue mm_percpu_wq: flags=0x8 [ 2392.746577] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2392.753793] pending: vmstat_update [ 2392.755329] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2392.762564] pending: vmstat_update [ 2392.769938] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2392.775691] pending: vmstat_update [ 2392.777576] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2392.782353] pending: vmstat_update [ 2392.784323] workqueue writeback: flags=0x4a [ 2392.786589] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2392.797381] pending: wb_workfn, wb_workfn [ 2392.803554] workqueue xprtiod: flags=0xa [ 2392.807937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2392.817509] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2392.827019] workqueue ipv6_addrconf: flags=0x40008 [ 2392.828704] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2392.835680] pending: addrconf_verify_work [ 2392.836987] workqueue knbd0-recv: flags=0x1a [ 2392.847847] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2392.852419] in-flight: 647:recv_work [ 2392.853495] workqueue ptlrpc_pinger: flags=0x2 [ 2392.855184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2392.857668] pending: ptlrpc_pinger_main [ptlrpc] [ 2392.860058] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2382s workers=2 idle: 652 [ 2400.061190] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 2400.061260] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 2400.067362] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2400.067440] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2400.072828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2400.072909] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2400.072916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2400.072919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2400.072932] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2400.072937] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2400.072941] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2400.072943] RDX: 0000000000007b21 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2400.072946] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2400.072948] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2400.072950] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2400.072953] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2400.072956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.072958] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2400.072966] Call Trace: [ 2400.072970] [ 2400.072976] ? show_regs.cold.9+0x22/0x2f [ 2400.072984] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2400.072990] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2400.072993] ? watchdog+0x40/0x40 [ 2400.072998] ? hrtimer_interrupt+0x128/0x2c0 [ 2400.073002] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2400.073009] ? apic_timer_interrupt+0xf/0x20 [ 2400.073013] [ 2400.073016] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2400.073020] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2400.073024] do_raw_spin_lock+0x177/0x190 [ 2400.073029] _raw_spin_lock+0x21/0x30 [ 2400.073033] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2400.073330] obd_mod_free+0x46/0x140 [ptlrpc] [ 2400.073528] mdc_commit_open+0x91/0xb0 [mdc] [ 2400.073563] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2400.073770] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2400.073967] after_reply+0xe2c/0x1780 [ptlrpc] [ 2400.074169] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2400.074372] ? __next_timer_interrupt+0x160/0x160 [ 2400.074382] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2400.108495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2400.108506] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2400.108521] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2400.108525] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2400.108531] RAX: 000000000000651e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2400.108534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2400.108536] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2400.108538] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2400.108540] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2400.108544] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2400.108546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.108549] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2400.108557] Call Trace: [ 2400.108563] [ 2400.108567] ? show_regs.cold.9+0x22/0x2f [ 2400.108576] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2400.108582] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2400.108587] ? watchdog+0x40/0x40 [ 2400.108591] ? hrtimer_interrupt+0x128/0x2c0 [ 2400.108595] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2400.108603] ? apic_timer_interrupt+0xf/0x20 [ 2400.108608] [ 2400.108611] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2400.108618] do_raw_spin_lock+0x177/0x190 [ 2400.108622] _raw_spin_lock+0x21/0x30 [ 2400.108627] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2400.110125] ? do_raw_spin_unlock+0x75/0x190 [ 2400.110134] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2400.110336] ? do_wait_intr+0xf0/0xf0 [ 2400.110344] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2400.111752] mdc_close+0x53b/0xd70 [mdc] [ 2400.111789] lmv_close+0x18d/0x490 [lmv] [ 2400.111817] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2400.111922] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2400.112005] ll_md_close+0x274/0x800 [lustre] [ 2400.112089] ll_file_release+0x128/0xc50 [lustre] [ 2400.112177] __fput+0x10d/0x440 [ 2400.112186] ____fput+0x12/0x20 [ 2400.112189] task_work_run+0xc6/0x110 [ 2400.112196] exit_to_usermode_loop+0x1dd/0x1f0 [ 2400.112203] do_syscall_64+0x3d6/0x440 [ 2400.112207] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2400.112215] RIP: 0033:0x7f1a63983cd8 [ 2400.112222] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2400.112226] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2400.112232] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2400.112234] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2400.112236] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2400.112238] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2400.112240] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2400.625339] ? do_wait_intr+0xf0/0xf0 [ 2400.627954] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2400.634987] kthread+0x1d1/0x200 [ 2400.638511] ? set_kthread_struct+0x70/0x70 [ 2400.643928] ret_from_fork+0x1f/0x30 [ 2416.057512] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 2416.062472] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2416.082250] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2416.087068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2416.089781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2416.091467] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2416.098614] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2416.101646] RAX: 0000000000003b4f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2416.104398] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2416.106934] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2416.110850] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2416.113296] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2416.119382] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2416.125889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2416.127454] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2416.133849] Call Trace: [ 2416.134632] [ 2416.135256] ? show_regs.cold.9+0x22/0x2f [ 2416.139575] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2416.140911] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2416.142237] ? watchdog+0x40/0x40 [ 2416.143340] ? hrtimer_interrupt+0x128/0x2c0 [ 2416.145294] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2416.148369] ? apic_timer_interrupt+0xf/0x20 [ 2416.149804] [ 2416.150432] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2416.152833] do_raw_spin_lock+0x177/0x190 [ 2416.154401] _raw_spin_lock+0x21/0x30 [ 2416.155692] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2416.157855] ? do_raw_spin_unlock+0x75/0x190 [ 2416.159386] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2416.162696] ? do_wait_intr+0xf0/0xf0 [ 2416.164202] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2416.166171] mdc_close+0x53b/0xd70 [mdc] [ 2416.167610] lmv_close+0x18d/0x490 [lmv] [ 2416.169013] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2416.170867] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2416.175983] ll_md_close+0x274/0x800 [lustre] [ 2416.177872] ll_file_release+0x128/0xc50 [lustre] [ 2416.183225] __fput+0x10d/0x440 [ 2416.184198] ____fput+0x12/0x20 [ 2416.185384] task_work_run+0xc6/0x110 [ 2416.186613] exit_to_usermode_loop+0x1dd/0x1f0 [ 2416.188324] do_syscall_64+0x3d6/0x440 [ 2416.189649] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2416.191365] RIP: 0033:0x7f1a63983cd8 [ 2416.192565] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2416.200210] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2416.205238] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2416.213445] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2416.220871] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2416.227050] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2416.232162] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2420.058193] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2420.065391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2420.117397] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2420.128121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2420.135164] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2420.140639] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2420.162342] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2420.172715] RAX: 0000000000005cc7 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2420.184720] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2420.195718] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2420.209790] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2420.217979] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2420.223026] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2420.229761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2420.233340] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2420.237754] Call Trace: [ 2420.242723] [ 2420.243995] ? show_regs.cold.9+0x22/0x2f [ 2420.247707] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2420.249900] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2420.251836] ? watchdog+0x40/0x40 [ 2420.253004] ? hrtimer_interrupt+0x128/0x2c0 [ 2420.254995] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2420.257613] ? apic_timer_interrupt+0xf/0x20 [ 2420.264772] [ 2420.266654] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2420.275504] do_raw_spin_lock+0x177/0x190 [ 2420.278771] _raw_spin_lock+0x21/0x30 [ 2420.280984] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2420.287027] ? do_raw_spin_unlock+0x75/0x190 [ 2420.290740] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2420.298520] ? do_wait_intr+0xf0/0xf0 [ 2420.302047] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2420.314735] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2420.320859] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2420.325305] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2420.332421] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2420.334364] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2420.339824] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2420.343070] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2420.352081] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2420.354339] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2420.361144] lmv_intent_open+0x338/0x1150 [lmv] [ 2420.368508] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2420.376888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2420.385382] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2420.391450] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2420.394920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2420.400817] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2420.404420] ? do_raw_spin_unlock+0x75/0x190 [ 2420.406540] ? _raw_spin_unlock+0x12/0x30 [ 2420.408504] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2420.410452] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2420.417739] ll_atomic_open+0x36a/0xd70 [lustre] [ 2420.424326] lookup_open+0x356/0xb40 [ 2420.427864] do_last+0x313/0xfc0 [ 2420.434828] ? nd_jump_root+0xe5/0x160 [ 2420.440884] ? path_init+0x437/0x520 [ 2420.442386] path_openat+0xf7/0x500 [ 2420.449040] ? do_raw_spin_unlock+0x75/0x190 [ 2420.454378] do_filp_open+0x99/0x140 [ 2420.462675] ? getname_flags+0x6e/0x330 [ 2420.464603] ? __check_object_size+0xff/0x256 [ 2420.470436] ? do_raw_spin_unlock+0x75/0x190 [ 2420.474140] ? _raw_spin_unlock+0x12/0x30 [ 2420.484838] do_sys_openat2+0x2b4/0x410 [ 2420.486244] do_sys_open+0x73/0xa0 [ 2420.492535] __x64_sys_openat+0x24/0x30 [ 2420.493962] do_syscall_64+0xc1/0x440 [ 2420.499745] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2420.512731] RIP: 0033:0x7f1a63983332 [ 2420.515469] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2420.541137] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2420.552324] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2420.555057] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2420.567061] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2420.572633] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2420.590723] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2423.265849] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 400s! [ 2423.273855] Showing busy workqueues and worker pools: [ 2423.275969] workqueue events: flags=0x0 [ 2423.280296] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2423.283386] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2423.290331] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2423.296897] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2423.312884] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2423.326126] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2423.343469] workqueue events_long: flags=0x0 [ 2423.344963] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2423.356608] pending: lu_objects_flush [obdclass] [ 2423.359829] workqueue events_unbound: flags=0x2 [ 2423.362820] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2423.369847] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2423.377890] workqueue events_power_efficient: flags=0x80 [ 2423.381200] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2423.385747] pending: neigh_periodic_work, do_cache_clean [ 2423.388898] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2423.397476] pending: neigh_periodic_work, check_lifetime [ 2423.401898] workqueue mm_percpu_wq: flags=0x8 [ 2423.404774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2423.407822] pending: vmstat_update [ 2423.412480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2423.415826] pending: vmstat_update [ 2423.417564] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2423.425813] pending: vmstat_update [ 2423.430504] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2423.439413] pending: vmstat_update [ 2423.444989] workqueue writeback: flags=0x4a [ 2423.448821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2423.457364] pending: wb_workfn, wb_workfn [ 2423.465165] workqueue xprtiod: flags=0xa [ 2423.471907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2423.479830] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2423.483361] workqueue ipv6_addrconf: flags=0x40008 [ 2423.494808] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2423.504276] pending: addrconf_verify_work [ 2423.516939] workqueue knbd0-recv: flags=0x1a [ 2423.526944] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2423.539881] in-flight: 647:recv_work [ 2423.541368] workqueue ptlrpc_pinger: flags=0x2 [ 2423.543616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2423.546962] pending: ptlrpc_pinger_main [ptlrpc] [ 2423.550357] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2413s workers=2 idle: 652 [ 2428.058128] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 2428.059491] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 2428.079189] Modules linked in: lustre(O) osp(O) [ 2428.088307] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2428.088466] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2428.088473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2428.088476] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2428.088490] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2428.088494] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2428.088499] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2428.088501] RDX: 0000000000004244 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2428.088503] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2428.088505] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2428.088508] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2428.088511] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2428.088513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2428.088515] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2428.088525] Call Trace: [ 2428.088531] [ 2428.088536] ? show_regs.cold.9+0x22/0x2f [ 2428.088545] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2428.088552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2428.088556] ? watchdog+0x40/0x40 [ 2428.088560] ? hrtimer_interrupt+0x128/0x2c0 [ 2428.088564] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2428.088572] ? apic_timer_interrupt+0xf/0x20 [ 2428.088577] [ 2428.088580] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2428.088585] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2428.088590] do_raw_spin_lock+0x177/0x190 [ 2428.088595] _raw_spin_lock+0x21/0x30 [ 2428.088600] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2428.091678] obd_mod_free+0x46/0x140 [ptlrpc] [ 2428.094885] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2428.094969] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2428.094976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2428.094980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.094994] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2428.094998] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2428.095504] RAX: 0000000000007f0a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2428.095507] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2428.095509] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2428.095511] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2428.095514] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2428.095518] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2428.095521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2428.095523] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2428.095533] Call Trace: [ 2428.095539] [ 2428.095547] ? show_regs.cold.9+0x22/0x2f [ 2428.095560] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2428.095568] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2428.095572] ? watchdog+0x40/0x40 [ 2428.095577] ? hrtimer_interrupt+0x128/0x2c0 [ 2428.095581] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2428.095590] ? apic_timer_interrupt+0xf/0x20 [ 2428.095596] [ 2428.095599] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.095607] do_raw_spin_lock+0x177/0x190 [ 2428.095612] _raw_spin_lock+0x21/0x30 [ 2428.095617] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2428.096121] ? do_raw_spin_unlock+0x75/0x190 [ 2428.096128] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2428.096557] ? do_wait_intr+0xf0/0xf0 [ 2428.096651] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2428.096845] mdc_close+0x53b/0xd70 [mdc] [ 2428.098447] lmv_close+0x18d/0x490 [lmv] [ 2428.098484] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2428.098610] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2428.098691] ll_md_close+0x274/0x800 [lustre] [ 2428.098774] ll_file_release+0x128/0xc50 [lustre] [ 2428.098855] __fput+0x10d/0x440 [ 2428.098863] ____fput+0x12/0x20 [ 2428.099471] task_work_run+0xc6/0x110 [ 2428.099480] exit_to_usermode_loop+0x1dd/0x1f0 [ 2428.099489] do_syscall_64+0x3d6/0x440 [ 2428.099494] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2428.099503] RIP: 0033:0x7f1a63983cd8 [ 2428.099511] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2428.099516] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2428.099522] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2428.099524] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2428.099526] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2428.099528] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2428.099530] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2428.449768] mdc_commit_open+0x91/0xb0 [mdc] [ 2428.452074] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2428.454733] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2428.459089] after_reply+0xe2c/0x1780 [ptlrpc] [ 2428.461709] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2428.464808] ? __next_timer_interrupt+0x160/0x160 [ 2428.467939] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2428.470464] ? do_wait_intr+0xf0/0xf0 [ 2428.472810] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2428.476990] kthread+0x1d1/0x200 [ 2428.480923] ? set_kthread_struct+0x70/0x70 [ 2428.483254] ret_from_fork+0x1f/0x30 [ 2442.422064] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 2442.432771] rcu: 3-...!: (242789 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 2442.440652] (detected by 1, t=420016 jiffies, g=269633, q=318) [ 2442.443567] Sending NMI from CPU 1 to CPUs 3: [ 2442.453872] NMI backtrace for cpu 3 [ 2442.453879] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2442.453881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2442.453883] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2442.453885] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2442.453888] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 2442.453890] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2442.453892] RDX: 0000000000003a6a RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2442.453893] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2442.453894] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2442.453896] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2442.453897] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2442.453899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2442.453900] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2442.453901] Call Trace: [ 2442.453902] [ 2442.453903] ? show_regs.cold.9+0x22/0x2f [ 2442.453904] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2442.453905] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2442.453906] ? nmi_handle+0xbd/0x210 [ 2442.453907] ? default_do_nmi+0x4b/0x1b0 [ 2442.453908] ? do_nmi+0x280/0x380 [ 2442.453909] ? end_repeat_nmi+0x16/0x1f [ 2442.453910] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2442.453912] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2442.453913] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2442.453914] [ 2442.453915] do_raw_spin_lock+0x177/0x190 [ 2442.453916] _raw_spin_lock+0x21/0x30 [ 2442.453917] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2442.453918] obd_mod_free+0x46/0x140 [ptlrpc] [ 2442.453919] mdc_commit_open+0x91/0xb0 [mdc] [ 2442.453920] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2442.453921] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2442.453922] after_reply+0xe2c/0x1780 [ptlrpc] [ 2442.453923] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2442.453925] ? __next_timer_interrupt+0x160/0x160 [ 2442.453926] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2442.453927] ? do_wait_intr+0xf0/0xf0 [ 2442.453928] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2442.453929] kthread+0x1d1/0x200 [ 2442.453930] ? set_kthread_struct+0x70/0x70 [ 2442.453931] ret_from_fork+0x1f/0x30 [ 2442.458058] rcu: rcu_sched kthread starved for 419662 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 2442.723594] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2442.731806] rcu: RCU grace-period kthread stack dump: [ 2442.734205] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2442.741479] Call Trace: [ 2442.743667] __schedule+0x351/0xcb0 [ 2442.747224] schedule+0xc0/0x180 [ 2442.754879] schedule_timeout+0xb4/0x190 [ 2442.756354] ? do_raw_spin_unlock+0x75/0x190 [ 2442.757749] ? __next_timer_interrupt+0x160/0x160 [ 2442.761829] ? prepare_to_swait_event+0xf3/0x140 [ 2442.765675] rcu_gp_kthread+0x8dd/0x15b0 [ 2442.768303] ? force_qs_rnp+0x2c0/0x2c0 [ 2442.771601] kthread+0x1d1/0x200 [ 2442.772782] ? set_kthread_struct+0x70/0x70 [ 2442.775193] ret_from_fork+0x1f/0x30 [ 2442.777670] rcu: Stack dump where RCU GP kthread last ran: [ 2442.784398] Sending NMI from CPU 1 to CPUs 0: [ 2442.788408] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 2444.061572] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 2444.072962] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2444.148720] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2444.167946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2444.181622] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2444.188180] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2444.228662] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2444.243321] RAX: 0000000000006a0c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2444.251506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2444.253953] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2444.271088] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2444.274190] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2444.284858] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2444.292701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2444.296823] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2444.299465] Call Trace: [ 2444.326457] [ 2444.327275] ? show_regs.cold.9+0x22/0x2f [ 2444.328746] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2444.339661] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2444.343690] ? watchdog+0x40/0x40 [ 2444.345883] ? hrtimer_interrupt+0x128/0x2c0 [ 2444.347395] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2444.351449] ? apic_timer_interrupt+0xf/0x20 [ 2444.354395] [ 2444.357918] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2444.361578] do_raw_spin_lock+0x177/0x190 [ 2444.363825] _raw_spin_lock+0x21/0x30 [ 2444.369039] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2444.376110] ? do_raw_spin_unlock+0x75/0x190 [ 2444.377687] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2444.382417] ? do_wait_intr+0xf0/0xf0 [ 2444.384586] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2444.390440] mdc_close+0x53b/0xd70 [mdc] [ 2444.392630] lmv_close+0x18d/0x490 [lmv] [ 2444.394195] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2444.401332] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2444.408762] ll_md_close+0x274/0x800 [lustre] [ 2444.416277] ll_file_release+0x128/0xc50 [lustre] [ 2444.420884] __fput+0x10d/0x440 [ 2444.423218] ____fput+0x12/0x20 [ 2444.425210] task_work_run+0xc6/0x110 [ 2444.431277] exit_to_usermode_loop+0x1dd/0x1f0 [ 2444.432871] do_syscall_64+0x3d6/0x440 [ 2444.438019] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2444.442860] RIP: 0033:0x7f1a63983cd8 [ 2444.444973] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2444.463580] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2444.467846] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2444.477598] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2444.486542] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2444.497172] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2444.503638] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2453.986083] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 430s! [ 2453.990552] Showing busy workqueues and worker pools: [ 2453.994054] workqueue events: flags=0x0 [ 2453.996252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2454.000031] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2454.004953] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2454.014708] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2454.028220] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2454.039367] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2454.050139] workqueue events_long: flags=0x0 [ 2454.053017] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2454.057021] pending: lu_objects_flush [obdclass] [ 2454.060596] workqueue events_unbound: flags=0x2 [ 2454.070597] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2454.074989] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2454.084862] workqueue events_power_efficient: flags=0x80 [ 2454.091094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2454.095328] pending: neigh_periodic_work, do_cache_clean [ 2454.103061] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2454.109274] pending: neigh_periodic_work, check_lifetime [ 2454.113381] workqueue mm_percpu_wq: flags=0x8 [ 2454.118787] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2454.124399] pending: vmstat_update [ 2454.134699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2454.137207] pending: vmstat_update [ 2454.148233] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2454.158072] pending: vmstat_update [ 2454.159502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2454.169935] pending: vmstat_update [ 2454.173402] workqueue writeback: flags=0x4a [ 2454.175596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2454.182904] pending: wb_workfn, wb_workfn [ 2454.184085] workqueue xprtiod: flags=0xa [ 2454.194819] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2454.204410] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2454.221447] workqueue ipv6_addrconf: flags=0x40008 [ 2454.226451] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2454.233290] pending: addrconf_verify_work [ 2454.242366] workqueue knbd0-recv: flags=0x1a [ 2454.249676] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2454.253554] in-flight: 647:recv_work [ 2454.255078] workqueue ptlrpc_pinger: flags=0x2 [ 2454.277383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2454.293283] pending: ptlrpc_pinger_main [ptlrpc] [ 2454.300537] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2444s workers=2 idle: 652 [ 2468.056023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2468.058014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2468.061448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2468.061767] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2468.061774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2468.061776] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2468.061790] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2468.061794] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2468.061799] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2468.061801] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2468.061803] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2468.061805] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2468.061808] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2468.061815] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2468.061818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2468.061820] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2468.061827] Call Trace: [ 2468.061843] [ 2468.061849] ? show_regs.cold.9+0x22/0x2f [ 2468.061860] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2468.061866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2468.061871] ? watchdog+0x40/0x40 [ 2468.061875] ? hrtimer_interrupt+0x128/0x2c0 [ 2468.061879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2468.062054] ? apic_timer_interrupt+0xf/0x20 [ 2468.062060] [ 2468.062063] ? native_safe_halt+0x2e/0x50 [ 2468.062067] kvm_wait+0xcd/0xe0 [ 2468.062074] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2468.062082] do_raw_spin_lock+0x177/0x190 [ 2468.062087] _raw_spin_lock+0x21/0x30 [ 2468.062096] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2468.068720] Modules linked in: lustre(O) [ 2468.112173] ? do_raw_spin_unlock+0x75/0x190 [ 2468.112192] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2468.120105] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2468.120481] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2468.120489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2468.120492] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2468.120505] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2468.120508] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2468.120513] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2468.120515] RDX: 00000000000017ef RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2468.120517] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2468.120519] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2468.120522] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2468.120525] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2468.120527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2468.120529] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2468.120598] Call Trace: [ 2468.120604] [ 2468.120610] ? show_regs.cold.9+0x22/0x2f [ 2468.120619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2468.120625] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2468.120630] ? watchdog+0x40/0x40 [ 2468.120634] ? hrtimer_interrupt+0x128/0x2c0 [ 2468.120638] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2468.120647] ? apic_timer_interrupt+0xf/0x20 [ 2468.120651] [ 2468.120654] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2468.120659] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2468.120664] do_raw_spin_lock+0x177/0x190 [ 2468.120669] _raw_spin_lock+0x21/0x30 [ 2468.120674] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2468.126948] ? do_wait_intr+0xf0/0xf0 [ 2468.128881] obd_mod_free+0x46/0x140 [ptlrpc] [ 2468.137428] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2468.140869] mdc_commit_open+0x91/0xb0 [mdc] [ 2468.144297] mdc_close+0x53b/0xd70 [mdc] [ 2468.146893] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2468.152540] lmv_close+0x18d/0x490 [lmv] [ 2468.152585] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2468.152693] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2468.152970] ll_md_close+0x274/0x800 [lustre] [ 2468.153067] ll_file_release+0x128/0xc50 [lustre] [ 2468.156886] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2468.160453] __fput+0x10d/0x440 [ 2468.164607] after_reply+0xe2c/0x1780 [ptlrpc] [ 2468.167566] ____fput+0x12/0x20 [ 2468.167587] task_work_run+0xc6/0x110 [ 2468.167596] exit_to_usermode_loop+0x1dd/0x1f0 [ 2468.167604] do_syscall_64+0x3d6/0x440 [ 2468.170788] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2468.171182] ? __next_timer_interrupt+0x160/0x160 [ 2468.171250] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2468.172705] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2468.173663] ? do_wait_intr+0xf0/0xf0 [ 2468.176138] RIP: 0033:0x7f1a63983cd8 [ 2468.176197] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2468.176203] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2468.176209] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2468.176211] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2468.176213] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2468.176216] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2468.176218] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2468.595089] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2468.597962] kthread+0x1d1/0x200 [ 2468.606158] ? set_kthread_struct+0x70/0x70 [ 2468.609466] ret_from_fork+0x1f/0x30 [ 2472.062235] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 2472.068527] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2472.099900] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2472.111639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2472.117657] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2472.121688] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2472.142036] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2472.149661] RAX: 0000000000007a84 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2472.157292] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2472.163288] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2472.171197] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2472.179077] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2472.185459] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2472.191757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2472.195096] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2472.198289] Call Trace: [ 2472.199199] [ 2472.199851] ? show_regs.cold.9+0x22/0x2f [ 2472.205034] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2472.207559] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2472.210166] ? watchdog+0x40/0x40 [ 2472.213966] ? hrtimer_interrupt+0x128/0x2c0 [ 2472.217866] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2472.220654] ? apic_timer_interrupt+0xf/0x20 [ 2472.223036] [ 2472.226406] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2472.229486] do_raw_spin_lock+0x177/0x190 [ 2472.231585] _raw_spin_lock+0x21/0x30 [ 2472.233346] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2472.236223] ? do_raw_spin_unlock+0x75/0x190 [ 2472.240086] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2472.242667] ? do_wait_intr+0xf0/0xf0 [ 2472.244819] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2472.248705] mdc_close+0x53b/0xd70 [mdc] [ 2472.250749] lmv_close+0x18d/0x490 [lmv] [ 2472.252725] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2472.256187] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2472.261660] ll_md_close+0x274/0x800 [lustre] [ 2472.266186] ll_file_release+0x128/0xc50 [lustre] [ 2472.269775] __fput+0x10d/0x440 [ 2472.272519] ____fput+0x12/0x20 [ 2472.273927] task_work_run+0xc6/0x110 [ 2472.277506] exit_to_usermode_loop+0x1dd/0x1f0 [ 2472.279557] do_syscall_64+0x3d6/0x440 [ 2472.282693] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2472.285572] RIP: 0033:0x7f1a63983cd8 [ 2472.288916] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2472.298570] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2472.302762] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2472.306259] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2472.309082] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2472.312574] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2472.316127] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2480.053017] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2480.057145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2480.111752] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2480.130729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2480.143744] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2480.152647] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2480.168359] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2480.177512] RAX: 0000000000007dd3 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2480.183787] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2480.187887] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2480.198259] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2480.206393] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2480.214470] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2480.226076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2480.229744] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2480.238473] Call Trace: [ 2480.240993] [ 2480.243740] ? show_regs.cold.9+0x22/0x2f [ 2480.249433] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2480.256810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2480.260864] ? watchdog+0x40/0x40 [ 2480.264543] ? hrtimer_interrupt+0x128/0x2c0 [ 2480.267750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2480.278470] ? apic_timer_interrupt+0xf/0x20 [ 2480.284754] [ 2480.286386] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2480.292079] do_raw_spin_lock+0x177/0x190 [ 2480.297608] _raw_spin_lock+0x21/0x30 [ 2480.301082] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2480.304671] ? do_raw_spin_unlock+0x75/0x190 [ 2480.309725] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2480.315887] ? do_wait_intr+0xf0/0xf0 [ 2480.317605] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2480.324310] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2480.333321] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2480.342979] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2480.344944] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2480.350405] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2480.358330] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2480.361237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2480.364175] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2480.365920] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2480.370917] lmv_intent_open+0x338/0x1150 [lmv] [ 2480.373237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2480.379054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2480.382135] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2480.385007] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2480.387435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2480.390458] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2480.393129] ? do_raw_spin_unlock+0x75/0x190 [ 2480.395439] ? _raw_spin_unlock+0x12/0x30 [ 2480.397656] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2480.401360] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2480.406887] ll_atomic_open+0x36a/0xd70 [lustre] [ 2480.410430] lookup_open+0x356/0xb40 [ 2480.414302] do_last+0x313/0xfc0 [ 2480.415887] ? nd_jump_root+0xe5/0x160 [ 2480.421844] ? path_init+0x437/0x520 [ 2480.423506] path_openat+0xf7/0x500 [ 2480.425468] ? do_raw_spin_unlock+0x75/0x190 [ 2480.429383] do_filp_open+0x99/0x140 [ 2480.431319] ? getname_flags+0x6e/0x330 [ 2480.433361] ? __check_object_size+0xff/0x256 [ 2480.435309] ? do_raw_spin_unlock+0x75/0x190 [ 2480.437582] ? _raw_spin_unlock+0x12/0x30 [ 2480.439000] do_sys_openat2+0x2b4/0x410 [ 2480.440132] do_sys_open+0x73/0xa0 [ 2480.441204] __x64_sys_openat+0x24/0x30 [ 2480.442501] do_syscall_64+0xc1/0x440 [ 2480.445269] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2480.448232] RIP: 0033:0x7f1a63983332 [ 2480.449616] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2480.457591] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2480.460577] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2480.463451] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2480.468848] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2480.472124] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2480.484260] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2484.709054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 461s! [ 2484.712318] Showing busy workqueues and worker pools: [ 2484.715278] workqueue events: flags=0x0 [ 2484.724922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2484.735085] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2484.740567] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2484.748761] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2484.757068] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2484.765973] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2484.772386] workqueue events_long: flags=0x0 [ 2484.776193] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2484.780120] pending: lu_objects_flush [obdclass] [ 2484.783120] workqueue events_unbound: flags=0x2 [ 2484.786034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2484.788813] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2484.793393] workqueue events_power_efficient: flags=0x80 [ 2484.796862] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2484.800651] pending: neigh_periodic_work, do_cache_clean [ 2484.804253] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2484.807889] pending: neigh_periodic_work, check_lifetime [ 2484.810454] workqueue mm_percpu_wq: flags=0x8 [ 2484.812319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2484.815433] pending: vmstat_update [ 2484.817783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2484.821547] pending: vmstat_update [ 2484.823408] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2484.826895] pending: vmstat_update [ 2484.829486] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2484.832919] pending: vmstat_update [ 2484.834564] workqueue writeback: flags=0x4a [ 2484.836791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2484.839438] pending: wb_workfn, wb_workfn [ 2484.841480] workqueue xprtiod: flags=0xa [ 2484.843366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2484.846390] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2484.850552] workqueue ipv6_addrconf: flags=0x40008 [ 2484.853267] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2484.856349] pending: addrconf_verify_work [ 2484.858501] workqueue knbd0-recv: flags=0x1a [ 2484.860557] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2484.862732] in-flight: 647:recv_work [ 2484.866426] workqueue ptlrpc_pinger: flags=0x2 [ 2484.868748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2484.871893] pending: ptlrpc_pinger_main [ptlrpc] [ 2484.875080] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2474s workers=2 idle: 652 [ 2496.057381] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2496.060662] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2496.062480] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2496.062554] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2496.062562] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2496.062565] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2496.062579] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2496.062584] RSP: 0018:ffffc38542073af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2496.062588] RAX: 0000000000007e8b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2496.062590] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2496.062593] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2496.062594] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2496.062597] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2496.062600] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2496.062602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2496.062605] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2496.062615] Call Trace: [ 2496.062621] [ 2496.062627] ? show_regs.cold.9+0x22/0x2f [ 2496.062636] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2496.062642] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2496.062646] ? watchdog+0x40/0x40 [ 2496.062650] ? hrtimer_interrupt+0x128/0x2c0 [ 2496.062654] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2496.062663] ? apic_timer_interrupt+0xf/0x20 [ 2496.062667] [ 2496.062670] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2496.062676] do_raw_spin_lock+0x177/0x190 [ 2496.062681] _raw_spin_lock+0x21/0x30 [ 2496.062687] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2496.063013] ? do_raw_spin_unlock+0x75/0x190 [ 2496.063019] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2496.063399] ? do_wait_intr+0xf0/0xf0 [ 2496.063407] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2496.063600] mdc_close+0x53b/0xd70 [mdc] [ 2496.063642] lmv_close+0x18d/0x490 [lmv] [ 2496.063676] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2496.063810] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2496.063951] ll_md_close+0x274/0x800 [lustre] [ 2496.064058] ll_file_release+0x128/0xc50 [lustre] [ 2496.064142] __fput+0x10d/0x440 [ 2496.064152] ____fput+0x12/0x20 [ 2496.064257] task_work_run+0xc6/0x110 [ 2496.064265] exit_to_usermode_loop+0x1dd/0x1f0 [ 2496.064274] do_syscall_64+0x3d6/0x440 [ 2496.064279] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2496.064285] RIP: 0033:0x7f1a63983cd8 [ 2496.064292] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2496.064296] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2496.064301] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2496.064303] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2496.064306] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2496.064308] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2496.064310] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2496.274596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2496.295580] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2496.300576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2496.304805] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2496.307766] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2496.316636] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2496.320484] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2496.323999] RDX: 0000000000007e8d RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2496.327465] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2496.330926] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2496.334479] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2496.337924] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2496.341933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2496.344914] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2496.347991] Call Trace: [ 2496.348979] [ 2496.349776] ? show_regs.cold.9+0x22/0x2f [ 2496.351853] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2496.353976] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2496.356318] ? watchdog+0x40/0x40 [ 2496.358179] ? hrtimer_interrupt+0x128/0x2c0 [ 2496.360372] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2496.362887] ? apic_timer_interrupt+0xf/0x20 [ 2496.365126] [ 2496.366162] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2496.369107] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2496.371627] do_raw_spin_lock+0x177/0x190 [ 2496.373321] _raw_spin_lock+0x21/0x30 [ 2496.375103] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2496.377812] obd_mod_free+0x46/0x140 [ptlrpc] [ 2496.380124] mdc_commit_open+0x91/0xb0 [mdc] [ 2496.382430] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2496.384742] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2496.390467] after_reply+0xe2c/0x1780 [ptlrpc] [ 2496.395484] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2496.399512] ? __next_timer_interrupt+0x160/0x160 [ 2496.401706] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2496.404544] ? do_wait_intr+0xf0/0xf0 [ 2496.406467] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2496.412513] kthread+0x1d1/0x200 [ 2496.414906] ? set_kthread_struct+0x70/0x70 [ 2496.416849] ret_from_fork+0x1f/0x30 [ 2500.057304] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2500.062476] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2500.113472] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2500.118581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2500.141631] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2500.146979] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2500.164014] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2500.175672] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2500.187022] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2500.196044] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2500.203086] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2500.208034] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2500.216214] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2500.222090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2500.227941] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2500.235932] Call Trace: [ 2500.238041] [ 2500.241504] ? show_regs.cold.9+0x22/0x2f [ 2500.246880] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2500.253382] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2500.255152] ? watchdog+0x40/0x40 [ 2500.256325] ? hrtimer_interrupt+0x128/0x2c0 [ 2500.262695] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2500.265533] ? apic_timer_interrupt+0xf/0x20 [ 2500.272137] [ 2500.278000] ? native_safe_halt+0x2e/0x50 [ 2500.279715] kvm_wait+0xcd/0xe0 [ 2500.280912] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2500.292143] do_raw_spin_lock+0x177/0x190 [ 2500.303221] _raw_spin_lock+0x21/0x30 [ 2500.310436] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2500.319828] ? do_raw_spin_unlock+0x75/0x190 [ 2500.323116] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2500.328750] ? do_wait_intr+0xf0/0xf0 [ 2500.330071] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2500.343269] mdc_close+0x53b/0xd70 [mdc] [ 2500.346457] lmv_close+0x18d/0x490 [lmv] [ 2500.352067] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2500.356871] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2500.361955] ll_md_close+0x274/0x800 [lustre] [ 2500.367969] ll_file_release+0x128/0xc50 [lustre] [ 2500.373415] __fput+0x10d/0x440 [ 2500.376810] ____fput+0x12/0x20 [ 2500.380408] task_work_run+0xc6/0x110 [ 2500.381922] exit_to_usermode_loop+0x1dd/0x1f0 [ 2500.390910] do_syscall_64+0x3d6/0x440 [ 2500.393627] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2500.398101] RIP: 0033:0x7f1a63983cd8 [ 2500.400011] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2500.416815] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2500.426610] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2500.434461] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2500.439029] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2500.443024] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2500.448381] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2512.053025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2512.057223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2512.081445] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.087238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.092312] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2512.095107] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2512.104140] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2512.107661] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2512.110905] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2512.113767] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2512.117038] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2512.120274] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2512.123346] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2512.127135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.129849] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2512.133128] Call Trace: [ 2512.134439] [ 2512.135455] ? show_regs.cold.9+0x22/0x2f [ 2512.137801] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.140832] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.142991] ? watchdog+0x40/0x40 [ 2512.144917] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.147117] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.149464] ? apic_timer_interrupt+0xf/0x20 [ 2512.151518] [ 2512.152646] ? native_safe_halt+0x2e/0x50 [ 2512.154556] kvm_wait+0xcd/0xe0 [ 2512.156140] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2512.158835] do_raw_spin_lock+0x177/0x190 [ 2512.160816] _raw_spin_lock+0x21/0x30 [ 2512.162749] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2512.166127] ? do_raw_spin_unlock+0x75/0x190 [ 2512.169073] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2512.172250] ? do_wait_intr+0xf0/0xf0 [ 2512.176157] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2512.179852] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2512.184886] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2512.187979] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2512.191288] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2512.193506] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2512.195896] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2512.197338] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2512.198910] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2512.202549] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2512.210275] lmv_intent_open+0x338/0x1150 [lmv] [ 2512.215551] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2512.220515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2512.222750] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2512.227503] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2512.231495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2512.235117] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2512.238621] ? do_raw_spin_unlock+0x75/0x190 [ 2512.241854] ? _raw_spin_unlock+0x12/0x30 [ 2512.243894] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2512.247469] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2512.251125] ll_atomic_open+0x36a/0xd70 [lustre] [ 2512.254274] lookup_open+0x356/0xb40 [ 2512.256823] do_last+0x313/0xfc0 [ 2512.257709] ? nd_jump_root+0xe5/0x160 [ 2512.258631] ? path_init+0x437/0x520 [ 2512.259495] path_openat+0xf7/0x500 [ 2512.265751] ? do_raw_spin_unlock+0x75/0x190 [ 2512.269270] do_filp_open+0x99/0x140 [ 2512.272081] ? getname_flags+0x6e/0x330 [ 2512.274259] ? __check_object_size+0xff/0x256 [ 2512.277163] ? do_raw_spin_unlock+0x75/0x190 [ 2512.280495] ? _raw_spin_unlock+0x12/0x30 [ 2512.283256] do_sys_openat2+0x2b4/0x410 [ 2512.286055] do_sys_open+0x73/0xa0 [ 2512.288116] __x64_sys_openat+0x24/0x30 [ 2512.292243] do_syscall_64+0xc1/0x440 [ 2512.293157] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2512.296089] RIP: 0033:0x7f1a63983332 [ 2512.297829] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2512.310874] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2512.313817] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2512.320788] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2512.324021] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2512.326642] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2512.330913] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2514.913050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 491s! [ 2514.923446] Showing busy workqueues and worker pools: [ 2514.926356] workqueue events: flags=0x0 [ 2514.933068] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2514.938401] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2514.943439] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2514.951438] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2514.962106] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2514.973644] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2514.981776] workqueue events_long: flags=0x0 [ 2514.988479] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2514.994190] pending: lu_objects_flush [obdclass] [ 2515.004020] workqueue events_unbound: flags=0x2 [ 2515.008098] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2515.017858] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2515.030488] workqueue events_power_efficient: flags=0x80 [ 2515.038941] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2515.042419] pending: neigh_periodic_work, do_cache_clean [ 2515.044838] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2515.055408] pending: neigh_periodic_work, check_lifetime [ 2515.061662] workqueue mm_percpu_wq: flags=0x8 [ 2515.063400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2515.069368] pending: vmstat_update [ 2515.075197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2515.084259] pending: vmstat_update [ 2515.087347] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2515.094926] pending: vmstat_update [ 2515.099510] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2515.105166] pending: vmstat_update [ 2515.110014] workqueue writeback: flags=0x4a [ 2515.116910] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2515.123578] pending: wb_workfn, wb_workfn [ 2515.125987] workqueue xprtiod: flags=0xa [ 2515.127602] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2515.132468] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2515.136738] workqueue ipv6_addrconf: flags=0x40008 [ 2515.139433] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2515.143551] pending: addrconf_verify_work [ 2515.145684] workqueue knbd0-recv: flags=0x1a [ 2515.147855] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2515.153419] in-flight: 647:recv_work [ 2515.155494] workqueue ptlrpc_pinger: flags=0x2 [ 2515.159256] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2515.165052] pending: ptlrpc_pinger_main [ptlrpc] [ 2515.172811] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2505s workers=2 idle: 652 [ 2524.059466] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2524.059511] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2524.070007] Modules linked in: lustre(O) osp(O) [ 2524.080713] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2524.080785] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2524.080791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2524.080794] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2524.080808] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2524.080812] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2524.080815] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2524.080817] RDX: 000000000000167c RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2524.080819] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2524.080821] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2524.080823] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2524.080825] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2524.080827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2524.080829] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2524.080836] Call Trace: [ 2524.080841] [ 2524.080846] ? show_regs.cold.9+0x22/0x2f [ 2524.080856] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2524.080863] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2524.080868] ? watchdog+0x40/0x40 [ 2524.080872] ? hrtimer_interrupt+0x128/0x2c0 [ 2524.080876] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2524.080886] ? apic_timer_interrupt+0xf/0x20 [ 2524.080890] [ 2524.080893] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2524.080897] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2524.080901] do_raw_spin_lock+0x177/0x190 [ 2524.080915] _raw_spin_lock+0x21/0x30 [ 2524.080921] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2524.081518] obd_mod_free+0x46/0x140 [ptlrpc] [ 2524.081710] mdc_commit_open+0x91/0xb0 [mdc] [ 2524.081745] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2524.086927] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr [ 2524.116579] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2524.116847] after_reply+0xe2c/0x1780 [ptlrpc] [ 2524.117032] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2524.117224] ? __next_timer_interrupt+0x160/0x160 [ 2524.117234] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2524.127553] i2c_piix4 squashfs [ 2524.133741] ? do_wait_intr+0xf0/0xf0 [ 2524.133763] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2524.134190] kthread+0x1d1/0x200 [ 2524.134196] ? set_kthread_struct+0x70/0x70 [ 2524.134201] ret_from_fork+0x1f/0x30 [ 2524.384044] crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2524.407785] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2524.419241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2524.432868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2524.437226] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2524.459882] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2524.467330] RAX: 0000000000001186 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2524.475342] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2524.481093] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2524.487200] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2524.503042] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2524.514080] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2524.532254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2524.537724] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2524.544111] Call Trace: [ 2524.546055] [ 2524.546926] ? show_regs.cold.9+0x22/0x2f [ 2524.552300] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2524.557406] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2524.561702] ? watchdog+0x40/0x40 [ 2524.565333] ? hrtimer_interrupt+0x128/0x2c0 [ 2524.567090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2524.571824] ? apic_timer_interrupt+0xf/0x20 [ 2524.577941] [ 2524.578894] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2524.586884] do_raw_spin_lock+0x177/0x190 [ 2524.589846] _raw_spin_lock+0x21/0x30 [ 2524.595242] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2524.601266] ? do_raw_spin_unlock+0x75/0x190 [ 2524.605249] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2524.608279] ? do_wait_intr+0xf0/0xf0 [ 2524.612178] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2524.621564] mdc_close+0x53b/0xd70 [mdc] [ 2524.626398] lmv_close+0x18d/0x490 [lmv] [ 2524.628904] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2524.633603] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2524.643350] ll_md_close+0x274/0x800 [lustre] [ 2524.646590] ll_file_release+0x128/0xc50 [lustre] [ 2524.651630] __fput+0x10d/0x440 [ 2524.653040] ____fput+0x12/0x20 [ 2524.654836] task_work_run+0xc6/0x110 [ 2524.657020] exit_to_usermode_loop+0x1dd/0x1f0 [ 2524.659397] do_syscall_64+0x3d6/0x440 [ 2524.661603] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2524.663973] RIP: 0033:0x7f1a63983cd8 [ 2524.666322] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2524.676777] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2524.680693] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2524.685328] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2524.689286] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2524.693372] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2524.699793] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2528.057425] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2528.064587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2528.089127] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2528.097097] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2528.104822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2528.107231] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2528.122584] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2528.129606] RAX: 00000000000076f1 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2528.136650] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2528.142037] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2528.148615] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2528.154940] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2528.159832] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2528.165630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2528.171308] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2528.175832] Call Trace: [ 2528.178570] [ 2528.179823] ? show_regs.cold.9+0x22/0x2f [ 2528.183468] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2528.187639] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2528.191937] ? watchdog+0x40/0x40 [ 2528.194158] ? hrtimer_interrupt+0x128/0x2c0 [ 2528.195987] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2528.197932] ? apic_timer_interrupt+0xf/0x20 [ 2528.202070] [ 2528.205160] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2528.211501] do_raw_spin_lock+0x177/0x190 [ 2528.213958] _raw_spin_lock+0x21/0x30 [ 2528.219534] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2528.225492] ? do_raw_spin_unlock+0x75/0x190 [ 2528.228128] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2528.234874] ? do_wait_intr+0xf0/0xf0 [ 2528.240439] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2528.244070] mdc_close+0x53b/0xd70 [mdc] [ 2528.246005] lmv_close+0x18d/0x490 [lmv] [ 2528.251373] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2528.263707] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2528.271118] ll_md_close+0x274/0x800 [lustre] [ 2528.275183] ll_file_release+0x128/0xc50 [lustre] [ 2528.282111] __fput+0x10d/0x440 [ 2528.283321] ____fput+0x12/0x20 [ 2528.287989] task_work_run+0xc6/0x110 [ 2528.290587] exit_to_usermode_loop+0x1dd/0x1f0 [ 2528.294778] do_syscall_64+0x3d6/0x440 [ 2528.297142] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2528.299112] RIP: 0033:0x7f1a63983cd8 [ 2528.300526] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2528.313339] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2528.321483] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2528.327976] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2528.333771] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2528.337854] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2528.343906] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2540.053028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2540.056083] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.107707] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.117758] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2540.123850] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2540.126548] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2540.153583] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2540.169736] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2540.174691] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2540.183103] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2540.187037] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2540.191200] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2540.195724] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2540.200797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.206997] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2540.211951] Call Trace: [ 2540.213536] [ 2540.214616] ? show_regs.cold.9+0x22/0x2f [ 2540.216905] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.219681] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.222519] ? watchdog+0x40/0x40 [ 2540.225748] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.229965] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.234913] ? apic_timer_interrupt+0xf/0x20 [ 2540.240527] [ 2540.241634] ? native_safe_halt+0x2e/0x50 [ 2540.243620] kvm_wait+0xcd/0xe0 [ 2540.246477] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2540.248715] do_raw_spin_lock+0x177/0x190 [ 2540.250145] _raw_spin_lock+0x21/0x30 [ 2540.255560] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2540.260531] ? do_raw_spin_unlock+0x75/0x190 [ 2540.266004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2540.270004] ? do_wait_intr+0xf0/0xf0 [ 2540.271823] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2540.285073] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2540.291050] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2540.298987] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.305675] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2540.312934] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2540.318460] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2540.321667] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.327053] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2540.329401] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2540.332000] lmv_intent_open+0x338/0x1150 [lmv] [ 2540.336417] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.338824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.341924] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2540.343947] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2540.346412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.348637] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2540.350786] ? do_raw_spin_unlock+0x75/0x190 [ 2540.352562] ? _raw_spin_unlock+0x12/0x30 [ 2540.357226] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2540.359711] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2540.363215] ll_atomic_open+0x36a/0xd70 [lustre] [ 2540.367879] lookup_open+0x356/0xb40 [ 2540.370796] do_last+0x313/0xfc0 [ 2540.372865] ? nd_jump_root+0xe5/0x160 [ 2540.377244] ? path_init+0x437/0x520 [ 2540.379210] path_openat+0xf7/0x500 [ 2540.381201] ? do_raw_spin_unlock+0x75/0x190 [ 2540.389064] do_filp_open+0x99/0x140 [ 2540.391071] ? getname_flags+0x6e/0x330 [ 2540.394554] ? __check_object_size+0xff/0x256 [ 2540.397362] ? do_raw_spin_unlock+0x75/0x190 [ 2540.399873] ? _raw_spin_unlock+0x12/0x30 [ 2540.402802] do_sys_openat2+0x2b4/0x410 [ 2540.407416] do_sys_open+0x73/0xa0 [ 2540.410260] __x64_sys_openat+0x24/0x30 [ 2540.412181] do_syscall_64+0xc1/0x440 [ 2540.415202] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2540.420633] RIP: 0033:0x7f1a63983332 [ 2540.423210] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2540.435522] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2540.447644] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2540.454173] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2540.460375] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2540.465630] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2540.468572] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2545.632658] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 522s! [ 2545.636280] Showing busy workqueues and worker pools: [ 2545.647473] workqueue events: flags=0x0 [ 2545.649669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2545.661756] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2545.682343] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2545.686722] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2545.698027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2545.702741] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2545.709132] workqueue events_long: flags=0x0 [ 2545.711479] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2545.717866] pending: lu_objects_flush [obdclass] [ 2545.724292] workqueue events_unbound: flags=0x2 [ 2545.728608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2545.731056] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2545.736202] workqueue events_power_efficient: flags=0x80 [ 2545.738824] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2545.741829] pending: neigh_periodic_work, do_cache_clean [ 2545.747925] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2545.753844] pending: neigh_periodic_work, check_lifetime [ 2545.756229] workqueue mm_percpu_wq: flags=0x8 [ 2545.762921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2545.769298] pending: vmstat_update [ 2545.770631] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2545.777088] pending: vmstat_update [ 2545.779834] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2545.783274] pending: vmstat_update [ 2545.784529] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2545.787583] pending: vmstat_update [ 2545.789394] workqueue writeback: flags=0x4a [ 2545.791427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2545.794142] pending: wb_workfn, wb_workfn [ 2545.795571] workqueue xprtiod: flags=0xa [ 2545.797311] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2545.799589] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2545.803070] workqueue ipv6_addrconf: flags=0x40008 [ 2545.805127] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2545.808103] pending: addrconf_verify_work [ 2545.810118] workqueue knbd0-recv: flags=0x1a [ 2545.811908] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2545.814812] in-flight: 647:recv_work [ 2545.816457] workqueue ptlrpc_pinger: flags=0x2 [ 2545.818416] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2545.821115] pending: ptlrpc_pinger_main [ptlrpc] [ 2545.823574] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2535s workers=2 idle: 652 [ 2552.056046] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2552.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2552.060774] Modules linked in: [ 2552.067761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2552.067837] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2552.067844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2552.067848] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2552.067862] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2552.067867] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2552.067872] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2552.067874] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2552.067876] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2552.067878] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2552.067881] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2552.067884] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2552.067886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2552.067888] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2552.067898] Call Trace: [ 2552.067903] [ 2552.067909] ? show_regs.cold.9+0x22/0x2f [ 2552.067920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2552.067927] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2552.067931] ? watchdog+0x40/0x40 [ 2552.067935] ? hrtimer_interrupt+0x128/0x2c0 [ 2552.067939] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2552.067946] ? apic_timer_interrupt+0xf/0x20 [ 2552.067951] [ 2552.067954] ? native_safe_halt+0x2e/0x50 [ 2552.067958] kvm_wait+0xcd/0xe0 [ 2552.067965] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2552.067973] do_raw_spin_lock+0x177/0x190 [ 2552.067977] _raw_spin_lock+0x21/0x30 [ 2552.067982] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2552.072450] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2552.072530] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2552.072537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2552.072541] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2552.072554] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2552.072558] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2552.072562] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2552.072565] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2552.072567] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2552.072569] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2552.072571] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2552.072575] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2552.072577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2552.072580] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2552.072588] Call Trace: [ 2552.072593] [ 2552.072598] ? show_regs.cold.9+0x22/0x2f [ 2552.072607] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2552.072614] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2552.072618] ? watchdog+0x40/0x40 [ 2552.072622] ? hrtimer_interrupt+0x128/0x2c0 [ 2552.072626] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2552.072632] ? apic_timer_interrupt+0xf/0x20 [ 2552.072637] [ 2552.072639] ? native_safe_halt+0x2e/0x50 [ 2552.072643] kvm_wait+0xcd/0xe0 [ 2552.072650] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2552.072657] do_raw_spin_lock+0x177/0x190 [ 2552.072661] _raw_spin_lock+0x21/0x30 [ 2552.072665] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2552.097812] obd_mod_free+0x46/0x140 [ptlrpc] [ 2552.102621] ? do_raw_spin_unlock+0x75/0x190 [ 2552.102640] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2552.102898] ? do_wait_intr+0xf0/0xf0 [ 2552.102904] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2552.103079] mdc_close+0x53b/0xd70 [mdc] [ 2552.103110] lmv_close+0x18d/0x490 [lmv] [ 2552.103131] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2552.106558] mdc_commit_open+0x91/0xb0 [mdc] [ 2552.106600] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2552.106846] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2552.107034] after_reply+0xe2c/0x1780 [ptlrpc] [ 2552.108794] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2552.108900] ll_md_close+0x274/0x800 [lustre] [ 2552.108979] ll_file_release+0x128/0xc50 [lustre] [ 2552.109055] __fput+0x10d/0x440 [ 2552.109064] ____fput+0x12/0x20 [ 2552.109068] task_work_run+0xc6/0x110 [ 2552.109076] exit_to_usermode_loop+0x1dd/0x1f0 [ 2552.109082] do_syscall_64+0x3d6/0x440 [ 2552.109087] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2552.109095] RIP: 0033:0x7f1a63983cd8 [ 2552.109102] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2552.109106] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2552.109110] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2552.109112] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2552.109115] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2552.109117] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2552.109119] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2552.253250] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2552.255354] ? __next_timer_interrupt+0x160/0x160 [ 2552.257392] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2552.259345] ? do_wait_intr+0xf0/0xf0 [ 2552.260895] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2552.263613] kthread+0x1d1/0x200 [ 2552.266379] ? set_kthread_struct+0x70/0x70 [ 2552.268434] ret_from_fork+0x1f/0x30 [ 2556.057094] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2556.061032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2556.092619] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2556.107824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2556.118579] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2556.121670] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2556.139069] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2556.146529] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2556.153922] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2556.159317] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2556.167504] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2556.173717] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2556.180869] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2556.189828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2556.194859] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2556.201547] Call Trace: [ 2556.204565] [ 2556.206847] ? show_regs.cold.9+0x22/0x2f [ 2556.214798] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2556.216365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2556.220499] ? watchdog+0x40/0x40 [ 2556.224217] ? hrtimer_interrupt+0x128/0x2c0 [ 2556.226895] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2556.231813] ? apic_timer_interrupt+0xf/0x20 [ 2556.240435] [ 2556.241375] ? native_safe_halt+0x2e/0x50 [ 2556.248428] kvm_wait+0xcd/0xe0 [ 2556.257255] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2556.262762] do_raw_spin_lock+0x177/0x190 [ 2556.274436] _raw_spin_lock+0x21/0x30 [ 2556.275878] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2556.281694] ? do_raw_spin_unlock+0x75/0x190 [ 2556.283225] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2556.286741] ? do_wait_intr+0xf0/0xf0 [ 2556.289649] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2556.292613] mdc_close+0x53b/0xd70 [mdc] [ 2556.296455] lmv_close+0x18d/0x490 [lmv] [ 2556.300486] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2556.306639] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2556.309583] ll_md_close+0x274/0x800 [lustre] [ 2556.311863] ll_file_release+0x128/0xc50 [lustre] [ 2556.314613] __fput+0x10d/0x440 [ 2556.315998] ____fput+0x12/0x20 [ 2556.317441] task_work_run+0xc6/0x110 [ 2556.319384] exit_to_usermode_loop+0x1dd/0x1f0 [ 2556.321063] do_syscall_64+0x3d6/0x440 [ 2556.322826] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2556.325408] RIP: 0033:0x7f1a63983cd8 [ 2556.327305] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2556.335614] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2556.338782] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2556.342832] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2556.346643] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2556.350988] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2556.355818] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2572.055214] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2572.062873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2572.109050] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2572.118769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2572.124681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2572.129753] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2572.149287] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2572.152859] RAX: 0000000000007b11 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2572.157886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2572.162852] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2572.178907] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2572.182832] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2572.187895] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2572.197586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2572.202301] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2572.207597] Call Trace: [ 2572.212433] [ 2572.213730] ? show_regs.cold.9+0x22/0x2f [ 2572.216313] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2572.221362] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2572.224370] ? watchdog+0x40/0x40 [ 2572.225610] ? hrtimer_interrupt+0x128/0x2c0 [ 2572.236335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2572.239365] ? apic_timer_interrupt+0xf/0x20 [ 2572.241704] [ 2572.245592] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2572.250897] do_raw_spin_lock+0x177/0x190 [ 2572.253880] _raw_spin_lock+0x21/0x30 [ 2572.258581] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2572.262926] ? do_raw_spin_unlock+0x75/0x190 [ 2572.269571] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2572.275873] ? do_wait_intr+0xf0/0xf0 [ 2572.280050] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2572.285323] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2572.288911] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2572.295927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2572.301202] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2572.308311] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2572.313892] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2572.316485] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2572.323006] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2572.331065] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2572.340347] lmv_intent_open+0x338/0x1150 [lmv] [ 2572.347289] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2572.354598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2572.365450] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2572.370045] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2572.377141] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2572.386307] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2572.391727] ? do_raw_spin_unlock+0x75/0x190 [ 2572.395243] ? _raw_spin_unlock+0x12/0x30 [ 2572.399850] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2572.403221] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2572.408957] ll_atomic_open+0x36a/0xd70 [lustre] [ 2572.415926] lookup_open+0x356/0xb40 [ 2572.417333] do_last+0x313/0xfc0 [ 2572.429735] ? nd_jump_root+0xe5/0x160 [ 2572.437716] ? path_init+0x437/0x520 [ 2572.444759] path_openat+0xf7/0x500 [ 2572.450905] ? do_raw_spin_unlock+0x75/0x190 [ 2572.459713] do_filp_open+0x99/0x140 [ 2572.463197] ? getname_flags+0x6e/0x330 [ 2572.473420] ? __check_object_size+0xff/0x256 [ 2572.477786] ? do_raw_spin_unlock+0x75/0x190 [ 2572.488273] ? _raw_spin_unlock+0x12/0x30 [ 2572.497330] do_sys_openat2+0x2b4/0x410 [ 2572.505713] do_sys_open+0x73/0xa0 [ 2572.508461] __x64_sys_openat+0x24/0x30 [ 2572.514200] do_syscall_64+0xc1/0x440 [ 2572.517382] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2572.523652] RIP: 0033:0x7f1a63983332 [ 2572.526300] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2572.538447] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2572.550783] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2572.554642] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2572.557663] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2572.561843] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2572.566934] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2575.844644] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 552s! [ 2575.858241] Showing busy workqueues and worker pools: [ 2575.867549] workqueue events: flags=0x0 [ 2575.876464] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2575.882138] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2575.886402] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2575.892500] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2575.903728] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2575.910894] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2575.922200] workqueue events_long: flags=0x0 [ 2575.924569] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2575.936842] pending: lu_objects_flush [obdclass] [ 2575.942931] workqueue events_unbound: flags=0x2 [ 2575.946481] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2575.959524] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2575.970558] workqueue events_power_efficient: flags=0x80 [ 2575.974814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2575.985671] pending: neigh_periodic_work, do_cache_clean [ 2575.991721] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2575.997489] pending: neigh_periodic_work, check_lifetime [ 2576.003610] workqueue mm_percpu_wq: flags=0x8 [ 2576.008979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2576.015825] pending: vmstat_update [ 2576.023893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2576.035980] pending: vmstat_update [ 2576.041571] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2576.048749] pending: vmstat_update [ 2576.053678] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2576.063377] pending: vmstat_update [ 2576.070240] workqueue writeback: flags=0x4a [ 2576.073929] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2576.079823] pending: wb_workfn, wb_workfn [ 2576.082760] workqueue xprtiod: flags=0xa [ 2576.084809] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2576.087897] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2576.094041] workqueue ipv6_addrconf: flags=0x40008 [ 2576.099775] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2576.104244] pending: addrconf_verify_work [ 2576.106773] workqueue knbd0-recv: flags=0x1a [ 2576.108931] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2576.116634] in-flight: 647:recv_work [ 2576.124519] workqueue ptlrpc_pinger: flags=0x2 [ 2576.126173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2576.133177] pending: ptlrpc_pinger_main [ptlrpc] [ 2576.135896] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2566s workers=2 idle: 652 [ 2580.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2580.061669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2580.069148] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2580.085733] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2580.089257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2580.089332] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2580.089339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2580.089342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2580.089356] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2580.089360] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2580.089365] RAX: 0000000000001e2f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2580.089368] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2580.089370] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2580.089371] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2580.089374] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2580.089377] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2580.089380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2580.089382] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2580.089390] Call Trace: [ 2580.089395] [ 2580.089401] ? show_regs.cold.9+0x22/0x2f [ 2580.089411] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2580.089417] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2580.089421] ? watchdog+0x40/0x40 [ 2580.089426] ? hrtimer_interrupt+0x128/0x2c0 [ 2580.089429] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2580.089438] ? apic_timer_interrupt+0xf/0x20 [ 2580.089442] [ 2580.089445] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2580.089450] do_raw_spin_lock+0x177/0x190 [ 2580.089454] _raw_spin_lock+0x21/0x30 [ 2580.089458] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2580.089733] ? do_raw_spin_unlock+0x75/0x190 [ 2580.089739] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2580.090015] ? do_wait_intr+0xf0/0xf0 [ 2580.090024] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2580.090229] mdc_close+0x53b/0xd70 [mdc] [ 2580.090269] lmv_close+0x18d/0x490 [lmv] [ 2580.090297] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2580.090404] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2580.090486] ll_md_close+0x274/0x800 [lustre] [ 2580.090574] ll_file_release+0x128/0xc50 [lustre] [ 2580.090660] __fput+0x10d/0x440 [ 2580.090669] ____fput+0x12/0x20 [ 2580.090673] task_work_run+0xc6/0x110 [ 2580.090680] exit_to_usermode_loop+0x1dd/0x1f0 [ 2580.090687] do_syscall_64+0x3d6/0x440 [ 2580.090692] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2580.090698] RIP: 0033:0x7f1a63983cd8 [ 2580.090706] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2580.090709] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2580.090714] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2580.090716] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2580.090719] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2580.090721] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2580.090723] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2580.335151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2580.338350] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2580.339647] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2580.345199] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2580.347675] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2580.349429] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2580.351601] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2580.353916] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2580.355840] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2580.357842] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2580.360084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2580.361822] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2580.363781] Call Trace: [ 2580.364408] [ 2580.365451] ? show_regs.cold.9+0x22/0x2f [ 2580.367399] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2580.370206] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2580.372842] ? watchdog+0x40/0x40 [ 2580.375013] ? hrtimer_interrupt+0x128/0x2c0 [ 2580.377366] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2580.380251] ? apic_timer_interrupt+0xf/0x20 [ 2580.382736] [ 2580.383576] ? native_safe_halt+0x2e/0x50 [ 2580.385077] kvm_wait+0xcd/0xe0 [ 2580.387103] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2580.390126] do_raw_spin_lock+0x177/0x190 [ 2580.392412] _raw_spin_lock+0x21/0x30 [ 2580.394432] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2580.398394] obd_mod_free+0x46/0x140 [ptlrpc] [ 2580.401523] mdc_commit_open+0x91/0xb0 [mdc] [ 2580.403888] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2580.406890] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2580.410253] after_reply+0xe2c/0x1780 [ptlrpc] [ 2580.413166] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2580.415984] ? __next_timer_interrupt+0x160/0x160 [ 2580.418691] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2580.422972] ? do_wait_intr+0xf0/0xf0 [ 2580.427107] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2580.433020] kthread+0x1d1/0x200 [ 2580.436091] ? set_kthread_struct+0x70/0x70 [ 2580.439660] ret_from_fork+0x1f/0x30 [ 2584.057026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2584.062077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2584.094048] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2584.100831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2584.106208] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2584.109372] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2584.122209] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2584.127931] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2584.137233] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2584.141452] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2584.152467] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2584.159630] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2584.165006] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2584.170788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2584.175249] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2584.179765] Call Trace: [ 2584.181300] [ 2584.182650] ? show_regs.cold.9+0x22/0x2f [ 2584.185297] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2584.188066] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2584.190802] ? watchdog+0x40/0x40 [ 2584.192837] ? hrtimer_interrupt+0x128/0x2c0 [ 2584.196849] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2584.199779] ? apic_timer_interrupt+0xf/0x20 [ 2584.202845] [ 2584.204047] ? native_safe_halt+0x2e/0x50 [ 2584.206520] kvm_wait+0xcd/0xe0 [ 2584.208351] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2584.211462] do_raw_spin_lock+0x177/0x190 [ 2584.214430] _raw_spin_lock+0x21/0x30 [ 2584.217055] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2584.220204] ? do_raw_spin_unlock+0x75/0x190 [ 2584.221647] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2584.223176] ? do_wait_intr+0xf0/0xf0 [ 2584.225033] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2584.227459] mdc_close+0x53b/0xd70 [mdc] [ 2584.228989] lmv_close+0x18d/0x490 [lmv] [ 2584.230536] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2584.233229] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2584.235309] ll_md_close+0x274/0x800 [lustre] [ 2584.237496] ll_file_release+0x128/0xc50 [lustre] [ 2584.239507] __fput+0x10d/0x440 [ 2584.240383] ____fput+0x12/0x20 [ 2584.242042] task_work_run+0xc6/0x110 [ 2584.246053] exit_to_usermode_loop+0x1dd/0x1f0 [ 2584.248299] do_syscall_64+0x3d6/0x440 [ 2584.250228] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2584.251498] RIP: 0033:0x7f1a63983cd8 [ 2584.253533] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2584.263625] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2584.267411] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2584.270172] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2584.273193] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2584.277418] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2584.282315] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2604.053448] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 2604.056561] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2604.083084] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2604.090824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2604.095169] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2604.099819] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2604.119432] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2604.129206] RAX: 0000000000003797 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2604.140895] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2604.149911] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2604.154496] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2604.165663] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2604.173479] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2604.188560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2604.193323] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2604.197820] Call Trace: [ 2604.208762] [ 2604.210305] ? show_regs.cold.9+0x22/0x2f [ 2604.213144] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2604.215787] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2604.220261] ? watchdog+0x40/0x40 [ 2604.224323] ? hrtimer_interrupt+0x128/0x2c0 [ 2604.226491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2604.229471] ? apic_timer_interrupt+0xf/0x20 [ 2604.231115] [ 2604.233543] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2604.236739] do_raw_spin_lock+0x177/0x190 [ 2604.239781] _raw_spin_lock+0x21/0x30 [ 2604.243238] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2604.250014] ? do_raw_spin_unlock+0x75/0x190 [ 2604.255088] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2604.260818] ? do_wait_intr+0xf0/0xf0 [ 2604.263379] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2604.268494] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2604.282611] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2604.284906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2604.297376] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2604.305164] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2604.311120] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2604.314428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2604.317701] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2604.326178] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2604.329849] lmv_intent_open+0x338/0x1150 [lmv] [ 2604.335180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2604.339336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2604.348520] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2604.351562] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2604.355133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2604.358645] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2604.363635] ? do_raw_spin_unlock+0x75/0x190 [ 2604.367741] ? _raw_spin_unlock+0x12/0x30 [ 2604.371030] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2604.375634] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2604.380090] ll_atomic_open+0x36a/0xd70 [lustre] [ 2604.381691] lookup_open+0x356/0xb40 [ 2604.384121] do_last+0x313/0xfc0 [ 2604.386631] ? nd_jump_root+0xe5/0x160 [ 2604.390321] ? path_init+0x437/0x520 [ 2604.393705] path_openat+0xf7/0x500 [ 2604.395901] ? do_raw_spin_unlock+0x75/0x190 [ 2604.399058] do_filp_open+0x99/0x140 [ 2604.401651] ? getname_flags+0x6e/0x330 [ 2604.403498] ? __check_object_size+0xff/0x256 [ 2604.405067] ? do_raw_spin_unlock+0x75/0x190 [ 2604.408487] ? _raw_spin_unlock+0x12/0x30 [ 2604.410627] do_sys_openat2+0x2b4/0x410 [ 2604.412624] do_sys_open+0x73/0xa0 [ 2604.413818] __x64_sys_openat+0x24/0x30 [ 2604.415241] do_syscall_64+0xc1/0x440 [ 2604.416646] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2604.417983] RIP: 0033:0x7f1a63983332 [ 2604.424995] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2604.437367] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2604.441970] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2604.445035] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2604.451694] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2604.455503] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2604.460821] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2606.562178] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 583s! [ 2606.568154] Showing busy workqueues and worker pools: [ 2606.572387] workqueue events: flags=0x0 [ 2606.574585] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2606.577817] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2606.581505] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2606.585189] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2606.590271] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2606.593330] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2606.597822] workqueue events_long: flags=0x0 [ 2606.599558] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2606.602341] pending: lu_objects_flush [obdclass] [ 2606.604612] workqueue events_unbound: flags=0x2 [ 2606.607787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2606.611268] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2606.617501] workqueue events_power_efficient: flags=0x80 [ 2606.619772] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2606.622410] pending: neigh_periodic_work, do_cache_clean [ 2606.625149] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2606.627992] pending: neigh_periodic_work, check_lifetime [ 2606.630344] workqueue mm_percpu_wq: flags=0x8 [ 2606.632335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2606.635307] pending: vmstat_update [ 2606.637135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2606.640330] pending: vmstat_update [ 2606.642233] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2606.645174] pending: vmstat_update [ 2606.646883] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2606.649835] pending: vmstat_update [ 2606.651904] workqueue writeback: flags=0x4a [ 2606.653896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2606.656867] pending: wb_workfn, wb_workfn [ 2606.657922] workqueue xprtiod: flags=0xa [ 2606.658807] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2606.661935] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2606.666394] workqueue ipv6_addrconf: flags=0x40008 [ 2606.668587] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2606.671349] pending: addrconf_verify_work [ 2606.673976] workqueue knbd0-recv: flags=0x1a [ 2606.676571] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2606.679406] in-flight: 647:recv_work [ 2606.683104] workqueue ptlrpc_pinger: flags=0x2 [ 2606.685372] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2606.688049] pending: ptlrpc_pinger_main [ptlrpc] [ 2606.693383] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2596s workers=2 idle: 652 [ 2608.056057] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2608.058884] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2608.059797] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2608.096566] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2608.107115] Modules linked in: lustre(O) [ 2608.119204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2608.119220] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2608.119241] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2608.119246] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2608.119253] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2608.119255] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2608.119258] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2608.119260] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2608.119263] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2608.119267] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2608.119269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2608.119272] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2608.119280] Call Trace: [ 2608.119288] [ 2608.119296] ? show_regs.cold.9+0x22/0x2f [ 2608.119308] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2608.119314] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2608.119320] ? watchdog+0x40/0x40 [ 2608.119324] ? hrtimer_interrupt+0x128/0x2c0 [ 2608.119328] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2608.119335] ? apic_timer_interrupt+0xf/0x20 [ 2608.119339] [ 2608.119342] ? native_safe_halt+0x2e/0x50 [ 2608.119346] kvm_wait+0xcd/0xe0 [ 2608.119354] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2608.119364] do_raw_spin_lock+0x177/0x190 [ 2608.119368] _raw_spin_lock+0x21/0x30 [ 2608.119372] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2608.119993] ? do_raw_spin_unlock+0x75/0x190 [ 2608.120001] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2608.120328] ? do_wait_intr+0xf0/0xf0 [ 2608.120337] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2608.120827] mdc_close+0x53b/0xd70 [mdc] [ 2608.120952] lmv_close+0x18d/0x490 [lmv] [ 2608.120989] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2608.121184] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2608.121265] ll_md_close+0x274/0x800 [lustre] [ 2608.121344] ll_file_release+0x128/0xc50 [lustre] [ 2608.121421] __fput+0x10d/0x440 [ 2608.121432] ____fput+0x12/0x20 [ 2608.121436] task_work_run+0xc6/0x110 [ 2608.121446] exit_to_usermode_loop+0x1dd/0x1f0 [ 2608.121455] do_syscall_64+0x3d6/0x440 [ 2608.121460] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2608.121468] RIP: 0033:0x7f1a63983cd8 [ 2608.121477] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2608.121482] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2608.121487] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2608.121490] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2608.121493] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2608.121495] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2608.121497] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2608.418127] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2608.447547] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2608.460557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2608.471199] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2608.474183] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2608.491024] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2608.497596] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2608.505881] RDX: 0000000000001ee8 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2608.517754] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2608.529586] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2608.539013] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2608.543404] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2608.549957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2608.558879] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2608.564326] Call Trace: [ 2608.565410] [ 2608.566234] ? show_regs.cold.9+0x22/0x2f [ 2608.571602] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2608.574753] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2608.576495] ? watchdog+0x40/0x40 [ 2608.578285] ? hrtimer_interrupt+0x128/0x2c0 [ 2608.580660] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2608.585619] ? apic_timer_interrupt+0xf/0x20 [ 2608.587146] [ 2608.588855] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2608.590929] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2608.595216] do_raw_spin_lock+0x177/0x190 [ 2608.597808] _raw_spin_lock+0x21/0x30 [ 2608.599562] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2608.602833] obd_mod_free+0x46/0x140 [ptlrpc] [ 2608.613275] mdc_commit_open+0x91/0xb0 [mdc] [ 2608.616601] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2608.620372] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2608.630861] after_reply+0xe2c/0x1780 [ptlrpc] [ 2608.632870] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2608.641746] ? __next_timer_interrupt+0x160/0x160 [ 2608.644946] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2608.648720] ? do_wait_intr+0xf0/0xf0 [ 2608.653435] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2608.659403] kthread+0x1d1/0x200 [ 2608.662755] ? set_kthread_struct+0x70/0x70 [ 2608.666274] ret_from_fork+0x1f/0x30 [ 2612.057023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2612.059927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2612.083946] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2612.088056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2612.091991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2612.094367] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2612.105025] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2612.108114] RAX: 0000000000004f43 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2612.112671] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2612.116942] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2612.121921] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2612.125346] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2612.128194] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2612.131327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2612.133945] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2612.137142] Call Trace: [ 2612.138290] [ 2612.139278] ? show_regs.cold.9+0x22/0x2f [ 2612.141219] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2612.143258] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2612.145251] ? watchdog+0x40/0x40 [ 2612.146798] ? hrtimer_interrupt+0x128/0x2c0 [ 2612.148658] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2612.151506] ? apic_timer_interrupt+0xf/0x20 [ 2612.155661] [ 2612.157732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2612.159273] do_raw_spin_lock+0x177/0x190 [ 2612.160429] _raw_spin_lock+0x21/0x30 [ 2612.162916] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2612.183759] ? do_raw_spin_unlock+0x75/0x190 [ 2612.184930] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2612.186511] ? do_wait_intr+0xf0/0xf0 [ 2612.205192] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2612.214481] mdc_close+0x53b/0xd70 [mdc] [ 2612.220884] lmv_close+0x18d/0x490 [lmv] [ 2612.222073] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2612.228694] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2612.234314] ll_md_close+0x274/0x800 [lustre] [ 2612.238292] ll_file_release+0x128/0xc50 [lustre] [ 2612.242492] __fput+0x10d/0x440 [ 2612.246810] ____fput+0x12/0x20 [ 2612.250213] task_work_run+0xc6/0x110 [ 2612.254669] exit_to_usermode_loop+0x1dd/0x1f0 [ 2612.255991] do_syscall_64+0x3d6/0x440 [ 2612.260487] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2612.265547] RIP: 0033:0x7f1a63983cd8 [ 2612.268571] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2612.283563] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2612.291439] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2612.297329] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2612.302173] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2612.306679] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2612.312898] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2622.426591] rcu: INFO: rcu_sched self-detected stall on CPU [ 2622.429515] rcu: 3-...!: (349591 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 2622.434033] (t=600026 jiffies g=269633 q=379) [ 2622.436485] rcu: rcu_sched kthread starved for 599675 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 2622.441206] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2622.445875] rcu: RCU grace-period kthread stack dump: [ 2622.448000] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2622.452929] Call Trace: [ 2622.453928] __schedule+0x351/0xcb0 [ 2622.455589] schedule+0xc0/0x180 [ 2622.457313] schedule_timeout+0xb4/0x190 [ 2622.459428] ? do_raw_spin_unlock+0x75/0x190 [ 2622.461535] ? __next_timer_interrupt+0x160/0x160 [ 2622.463868] ? prepare_to_swait_event+0xf3/0x140 [ 2622.466237] rcu_gp_kthread+0x8dd/0x15b0 [ 2622.468188] ? force_qs_rnp+0x2c0/0x2c0 [ 2622.470300] kthread+0x1d1/0x200 [ 2622.471830] ? set_kthread_struct+0x70/0x70 [ 2622.473931] ret_from_fork+0x1f/0x30 [ 2622.475675] rcu: Stack dump where RCU GP kthread last ran: [ 2622.477973] Sending NMI from CPU 3 to CPUs 0: [ 2622.489169] NMI backtrace for cpu 0 [ 2622.489176] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2622.489178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2622.489181] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2622.489183] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2622.489185] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 [ 2622.489189] RAX: 0000000000007e13 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2622.489190] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2622.489192] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2622.490067] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2622.490072] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2622.490074] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2622.490075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2622.490076] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2622.490078] Call Trace: [ 2622.490079] [ 2622.490080] ? show_regs.cold.9+0x22/0x2f [ 2622.490081] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2622.490082] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2622.490083] ? nmi_handle+0xbd/0x210 [ 2622.490084] ? default_do_nmi+0x4b/0x1b0 [ 2622.490085] ? do_nmi+0x280/0x380 [ 2622.490086] ? end_repeat_nmi+0x16/0x1f [ 2622.490087] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2622.490088] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2622.490089] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2622.490090] [ 2622.490092] do_raw_spin_lock+0x177/0x190 [ 2622.490093] _raw_spin_lock+0x21/0x30 [ 2622.490094] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2622.490096] ? do_raw_spin_unlock+0x75/0x190 [ 2622.490097] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2622.490098] ? do_wait_intr+0xf0/0xf0 [ 2622.490099] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2622.490100] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2622.490102] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2622.490103] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2622.490104] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2622.490105] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2622.490106] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2622.490108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2622.490109] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2622.490110] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2622.490111] lmv_intent_open+0x338/0x1150 [lmv] [ 2622.490113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2622.490114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2622.490115] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2622.490116] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2622.490117] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2622.490119] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2622.490558] ? do_raw_spin_unlock+0x75/0x190 [ 2622.490560] ? _raw_spin_unlock+0x12/0x30 [ 2622.490562] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2622.490563] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2622.490564] ll_atomic_open+0x36a/0xd70 [lustre] [ 2622.490569] lookup_open+0x356/0xb40 [ 2622.490570] do_last+0x313/0xfc0 [ 2622.490571] ? nd_jump_root+0xe5/0x160 [ 2622.490572] ? path_init+0x437/0x520 [ 2622.490573] path_openat+0xf7/0x500 [ 2622.490574] ? do_raw_spin_unlock+0x75/0x190 [ 2622.490575] do_filp_open+0x99/0x140 [ 2622.490576] ? getname_flags+0x6e/0x330 [ 2622.490577] ? __check_object_size+0xff/0x256 [ 2622.490578] ? do_raw_spin_unlock+0x75/0x190 [ 2622.490580] ? _raw_spin_unlock+0x12/0x30 [ 2622.490581] do_sys_openat2+0x2b4/0x410 [ 2622.490582] do_sys_open+0x73/0xa0 [ 2622.490583] __x64_sys_openat+0x24/0x30 [ 2622.490584] do_syscall_64+0xc1/0x440 [ 2622.490585] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2622.490586] RIP: 0033:0x7f1a63983332 [ 2622.490587] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2622.490590] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2622.490594] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2622.490596] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2622.490597] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2622.490598] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2622.490600] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2622.490609] NMI backtrace for cpu 3 [ 2622.490616] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2622.723074] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2622.726695] Call Trace: [ 2622.727909] [ 2622.728986] ? dump_stack+0xbb/0x10e [ 2622.730513] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2622.733142] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2622.735259] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2622.737683] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2622.740701] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2622.743327] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2622.745621] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2622.748032] ? update_process_times+0x6d/0xc0 [ 2622.750140] ? tick_sched_handle.isra.18+0x27/0x90 [ 2622.752659] ? tick_sched_timer+0x53/0xb0 [ 2622.754934] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2622.757357] ? tick_sched_handle.isra.18+0x90/0x90 [ 2622.759678] ? hrtimer_interrupt+0x128/0x2c0 [ 2622.761986] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2622.764245] ? apic_timer_interrupt+0xf/0x20 [ 2622.766111] [ 2622.767165] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2622.769984] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2622.772327] ? do_raw_spin_lock+0x177/0x190 [ 2622.773870] ? _raw_spin_lock+0x21/0x30 [ 2622.775212] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2622.778002] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2622.780395] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2622.781948] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2622.784168] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2622.786983] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2622.789789] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2622.792300] ? __next_timer_interrupt+0x160/0x160 [ 2622.795034] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2622.797696] ? do_wait_intr+0xf0/0xf0 [ 2622.799592] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2622.802146] ? kthread+0x1d1/0x200 [ 2622.803912] ? set_kthread_struct+0x70/0x70 [ 2622.806603] ? ret_from_fork+0x1f/0x30 [ 2636.057080] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 2636.063425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2636.093305] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2636.096992] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2636.101009] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2636.102738] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2636.108388] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2636.110595] RAX: 000000000000778f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2636.112703] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2636.114759] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2636.116794] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2636.119011] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2636.121358] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2636.123871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2636.125502] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2636.128661] Call Trace: [ 2636.129327] [ 2636.129960] ? show_regs.cold.9+0x22/0x2f [ 2636.131107] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2636.132441] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2636.133752] ? watchdog+0x40/0x40 [ 2636.134848] ? hrtimer_interrupt+0x128/0x2c0 [ 2636.136383] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2636.137842] ? apic_timer_interrupt+0xf/0x20 [ 2636.139464] [ 2636.140199] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2636.141803] do_raw_spin_lock+0x177/0x190 [ 2636.143507] _raw_spin_lock+0x21/0x30 [ 2636.144834] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2636.146769] ? do_raw_spin_unlock+0x75/0x190 [ 2636.148081] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2636.149756] ? do_wait_intr+0xf0/0xf0 [ 2636.150963] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2636.152633] mdc_close+0x53b/0xd70 [mdc] [ 2636.153758] lmv_close+0x18d/0x490 [lmv] [ 2636.154983] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2636.157000] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2636.158565] ll_md_close+0x274/0x800 [lustre] [ 2636.159974] ll_file_release+0x128/0xc50 [lustre] [ 2636.161317] __fput+0x10d/0x440 [ 2636.162236] ____fput+0x12/0x20 [ 2636.163071] task_work_run+0xc6/0x110 [ 2636.164426] exit_to_usermode_loop+0x1dd/0x1f0 [ 2636.165717] do_syscall_64+0x3d6/0x440 [ 2636.166830] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2636.168285] RIP: 0033:0x7f1a63983cd8 [ 2636.169550] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2636.174839] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2636.176823] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2636.178996] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2636.180996] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2636.183522] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2636.185670] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2636.769558] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 613s! [ 2636.778066] Showing busy workqueues and worker pools: [ 2636.781994] workqueue events: flags=0x0 [ 2636.786449] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2636.789412] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2636.797925] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2636.800912] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2636.805954] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2636.809750] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2636.814375] workqueue events_long: flags=0x0 [ 2636.820478] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2636.824890] pending: lu_objects_flush [obdclass] [ 2636.830081] workqueue events_unbound: flags=0x2 [ 2636.832830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2636.835167] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2636.846579] workqueue events_power_efficient: flags=0x80 [ 2636.850232] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2636.864637] pending: neigh_periodic_work, do_cache_clean [ 2636.866735] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2636.873346] pending: neigh_periodic_work, check_lifetime [ 2636.882952] workqueue mm_percpu_wq: flags=0x8 [ 2636.888922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2636.894656] pending: vmstat_update [ 2636.897766] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2636.900065] pending: vmstat_update [ 2636.901275] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2636.905790] pending: vmstat_update [ 2636.908435] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2636.918176] pending: vmstat_update [ 2636.921139] workqueue writeback: flags=0x4a [ 2636.923035] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2636.925816] pending: wb_workfn, wb_workfn [ 2636.935714] workqueue xprtiod: flags=0xa [ 2636.937333] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2636.949230] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2636.952198] workqueue ipv6_addrconf: flags=0x40008 [ 2636.959033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2636.964138] pending: addrconf_verify_work [ 2636.973650] workqueue knbd0-recv: flags=0x1a [ 2636.975263] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2636.989109] in-flight: 647:recv_work [ 2636.990524] workqueue ptlrpc_pinger: flags=0x2 [ 2636.994473] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2636.998401] pending: ptlrpc_pinger_main [ptlrpc] [ 2637.000452] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2627s workers=2 idle: 652 [ 2640.058046] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2640.061012] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2640.080081] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2640.084956] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2640.088387] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2640.089991] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2640.105397] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2640.110619] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2640.113935] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2640.120742] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2640.127086] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2640.129192] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2640.142894] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2640.148405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2640.151631] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2640.159309] Call Trace: [ 2640.160585] [ 2640.161104] ? show_regs.cold.9+0x22/0x2f [ 2640.167775] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2640.169888] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2640.173622] ? watchdog+0x40/0x40 [ 2640.174840] ? hrtimer_interrupt+0x128/0x2c0 [ 2640.177147] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2640.179787] ? apic_timer_interrupt+0xf/0x20 [ 2640.183129] [ 2640.183710] ? native_safe_halt+0x2e/0x50 [ 2640.188447] kvm_wait+0xcd/0xe0 [ 2640.193094] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2640.195820] do_raw_spin_lock+0x177/0x190 [ 2640.199742] _raw_spin_lock+0x21/0x30 [ 2640.201574] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2640.204295] ? do_raw_spin_unlock+0x75/0x190 [ 2640.207032] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2640.212673] ? do_wait_intr+0xf0/0xf0 [ 2640.214237] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2640.219599] mdc_close+0x53b/0xd70 [mdc] [ 2640.221444] lmv_close+0x18d/0x490 [lmv] [ 2640.224713] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2640.231457] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2640.236584] ll_md_close+0x274/0x800 [lustre] [ 2640.239676] ll_file_release+0x128/0xc50 [lustre] [ 2640.242683] __fput+0x10d/0x440 [ 2640.247293] ____fput+0x12/0x20 [ 2640.249510] task_work_run+0xc6/0x110 [ 2640.251539] exit_to_usermode_loop+0x1dd/0x1f0 [ 2640.255525] do_syscall_64+0x3d6/0x440 [ 2640.258406] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2640.261719] RIP: 0033:0x7f1a63983cd8 [ 2640.264943] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2640.281190] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2640.284425] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2640.294210] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2640.298508] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2640.302399] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2640.307179] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2648.058126] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 2648.060381] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2648.076690] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2648.080796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2648.084131] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2648.085941] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2648.093591] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2648.096823] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2648.099469] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2648.102370] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2648.105189] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2648.108134] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2648.111078] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2648.114548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2648.116954] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2648.120074] Call Trace: [ 2648.121131] [ 2648.121937] ? show_regs.cold.9+0x22/0x2f [ 2648.124536] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2648.126628] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2648.128789] ? watchdog+0x40/0x40 [ 2648.130157] ? hrtimer_interrupt+0x128/0x2c0 [ 2648.131425] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2648.132698] ? apic_timer_interrupt+0xf/0x20 [ 2648.134419] [ 2648.135067] ? native_safe_halt+0x2e/0x50 [ 2648.136219] kvm_wait+0xcd/0xe0 [ 2648.137154] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2648.139414] do_raw_spin_lock+0x177/0x190 [ 2648.141051] _raw_spin_lock+0x21/0x30 [ 2648.142788] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2648.145191] obd_mod_free+0x46/0x140 [ptlrpc] [ 2648.147444] mdc_commit_open+0x91/0xb0 [mdc] [ 2648.149381] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2648.151725] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2648.154495] after_reply+0xe2c/0x1780 [ptlrpc] [ 2648.156652] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2648.158864] ? __next_timer_interrupt+0x160/0x160 [ 2648.160760] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2648.162835] ? do_wait_intr+0xf0/0xf0 [ 2648.164485] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2648.166769] kthread+0x1d1/0x200 [ 2648.168196] ? set_kthread_struct+0x70/0x70 [ 2648.169664] ret_from_fork+0x1f/0x30 [ 2664.053005] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 2664.058078] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 2664.074364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2664.074592] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2664.074598] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2664.074600] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2664.074613] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2664.074617] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2664.074621] RAX: 0000000000007c58 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2664.074624] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2664.074627] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2664.074628] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2664.074631] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2664.074634] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2664.074637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2664.074639] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2664.074647] Call Trace: [ 2664.074651] [ 2664.074656] ? show_regs.cold.9+0x22/0x2f [ 2664.074664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2664.074670] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2664.074674] ? watchdog+0x40/0x40 [ 2664.074678] ? hrtimer_interrupt+0x128/0x2c0 [ 2664.074682] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2664.074689] ? apic_timer_interrupt+0xf/0x20 [ 2664.074693] [ 2664.074696] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2664.074701] do_raw_spin_lock+0x177/0x190 [ 2664.074706] _raw_spin_lock+0x21/0x30 [ 2664.074710] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2664.075135] ? do_raw_spin_unlock+0x75/0x190 [ 2664.075141] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2664.075335] ? do_wait_intr+0xf0/0xf0 [ 2664.075344] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2664.075645] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2664.075872] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2664.076268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2664.076436] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2664.076527] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2664.076557] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2664.076582] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2664.076667] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2664.084639] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2664.084715] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2664.084722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2664.084725] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2664.084739] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2664.084742] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2664.084747] RAX: 0000000000007f85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2664.084749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2664.084752] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2664.084754] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2664.084756] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2664.084759] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2664.084762] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2664.084765] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2664.084773] Call Trace: [ 2664.084778] [ 2664.084783] ? show_regs.cold.9+0x22/0x2f [ 2664.084791] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2664.084798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2664.084802] ? watchdog+0x40/0x40 [ 2664.084807] ? hrtimer_interrupt+0x128/0x2c0 [ 2664.084811] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2664.084820] ? apic_timer_interrupt+0xf/0x20 [ 2664.084824] [ 2664.084827] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2664.084833] do_raw_spin_lock+0x177/0x190 [ 2664.084838] _raw_spin_lock+0x21/0x30 [ 2664.084844] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2664.121964] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2664.127500] ? do_raw_spin_unlock+0x75/0x190 [ 2664.132148] lmv_intent_open+0x338/0x1150 [lmv] [ 2664.132188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2664.132292] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2664.132371] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2664.132392] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2664.132483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2664.132571] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2664.132652] ? do_raw_spin_unlock+0x75/0x190 [ 2664.132660] ? _raw_spin_unlock+0x12/0x30 [ 2664.132665] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2664.132692] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2664.132712] ll_atomic_open+0x36a/0xd70 [lustre] [ 2664.132864] lookup_open+0x356/0xb40 [ 2664.132874] do_last+0x313/0xfc0 [ 2664.132879] ? nd_jump_root+0xe5/0x160 [ 2664.132883] ? path_init+0x437/0x520 [ 2664.132888] path_openat+0xf7/0x500 [ 2664.132903] ? do_raw_spin_unlock+0x75/0x190 [ 2664.132909] do_filp_open+0x99/0x140 [ 2664.132916] ? getname_flags+0x6e/0x330 [ 2664.132921] ? __check_object_size+0xff/0x256 [ 2664.132927] ? do_raw_spin_unlock+0x75/0x190 [ 2664.132931] ? _raw_spin_unlock+0x12/0x30 [ 2664.132936] do_sys_openat2+0x2b4/0x410 [ 2664.132941] do_sys_open+0x73/0xa0 [ 2664.132945] __x64_sys_openat+0x24/0x30 [ 2664.132948] do_syscall_64+0xc1/0x440 [ 2664.132954] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2664.132961] RIP: 0033:0x7f1a63983332 [ 2664.132968] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2664.132972] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2664.132978] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2664.132980] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2664.132982] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2664.132984] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2664.132985] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2664.541677] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2664.544645] ? do_wait_intr+0xf0/0xf0 [ 2664.547416] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2664.549997] mdc_close+0x53b/0xd70 [mdc] [ 2664.554064] lmv_close+0x18d/0x490 [lmv] [ 2664.556727] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2664.559606] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2664.561958] ll_md_close+0x274/0x800 [lustre] [ 2664.564420] ll_file_release+0x128/0xc50 [lustre] [ 2664.566699] __fput+0x10d/0x440 [ 2664.568502] ____fput+0x12/0x20 [ 2664.569820] task_work_run+0xc6/0x110 [ 2664.571564] exit_to_usermode_loop+0x1dd/0x1f0 [ 2664.573613] do_syscall_64+0x3d6/0x440 [ 2664.575026] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2664.577509] RIP: 0033:0x7f1a63983cd8 [ 2664.579375] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2664.587974] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2664.591602] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2664.596833] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2664.600007] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2664.609983] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2664.613962] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2667.489451] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 644s! [ 2667.491013] Showing busy workqueues and worker pools: [ 2667.492895] workqueue events: flags=0x0 [ 2667.495016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2667.497885] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2667.502359] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2667.505593] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2667.510622] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2667.514244] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2667.519575] workqueue events_long: flags=0x0 [ 2667.522014] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2667.524989] pending: lu_objects_flush [obdclass] [ 2667.527954] workqueue events_unbound: flags=0x2 [ 2667.531165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2667.535040] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2667.541139] workqueue events_power_efficient: flags=0x80 [ 2667.544887] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2667.547983] pending: neigh_periodic_work, do_cache_clean [ 2667.550904] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2667.554313] pending: neigh_periodic_work, check_lifetime [ 2667.556860] workqueue mm_percpu_wq: flags=0x8 [ 2667.559325] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2667.561900] pending: vmstat_update [ 2667.563630] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2667.566477] pending: vmstat_update [ 2667.568115] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2667.570908] pending: vmstat_update [ 2667.572758] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2667.575704] pending: vmstat_update [ 2667.577627] workqueue writeback: flags=0x4a [ 2667.579550] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2667.582245] pending: wb_workfn, wb_workfn [ 2667.583988] workqueue xprtiod: flags=0xa [ 2667.585694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2667.588014] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2667.591657] workqueue ipv6_addrconf: flags=0x40008 [ 2667.593976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2667.596765] pending: addrconf_verify_work [ 2667.598533] workqueue knbd0-recv: flags=0x1a [ 2667.600144] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2667.602787] in-flight: 647:recv_work [ 2667.604518] workqueue ptlrpc_pinger: flags=0x2 [ 2667.606324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2667.608848] pending: ptlrpc_pinger_main [ptlrpc] [ 2667.611228] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2657s workers=2 idle: 652 [ 2668.058777] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2668.070205] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2668.104778] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2668.110330] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2668.115644] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2668.123551] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2668.142329] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2668.150766] RAX: 0000000000007a91 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2668.158471] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2668.166728] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2668.174968] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2668.180830] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2668.186750] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2668.195761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2668.202673] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2668.206167] Call Trace: [ 2668.207570] [ 2668.208293] ? show_regs.cold.9+0x22/0x2f [ 2668.211188] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2668.213459] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2668.216094] ? watchdog+0x40/0x40 [ 2668.218136] ? hrtimer_interrupt+0x128/0x2c0 [ 2668.220144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2668.221744] ? apic_timer_interrupt+0xf/0x20 [ 2668.223754] [ 2668.224834] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2668.226788] do_raw_spin_lock+0x177/0x190 [ 2668.228314] _raw_spin_lock+0x21/0x30 [ 2668.229425] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2668.231237] ? do_raw_spin_unlock+0x75/0x190 [ 2668.232741] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2668.234307] ? do_wait_intr+0xf0/0xf0 [ 2668.235544] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2668.237397] mdc_close+0x53b/0xd70 [mdc] [ 2668.238931] lmv_close+0x18d/0x490 [lmv] [ 2668.240435] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2668.242197] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2668.243643] ll_md_close+0x274/0x800 [lustre] [ 2668.244948] ll_file_release+0x128/0xc50 [lustre] [ 2668.246551] __fput+0x10d/0x440 [ 2668.247680] ____fput+0x12/0x20 [ 2668.248860] task_work_run+0xc6/0x110 [ 2668.250489] exit_to_usermode_loop+0x1dd/0x1f0 [ 2668.252417] do_syscall_64+0x3d6/0x440 [ 2668.253640] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2668.255380] RIP: 0033:0x7f1a63983cd8 [ 2668.256554] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2668.262745] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2668.265176] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2668.268397] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2668.271199] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2668.272980] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2668.275231] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2676.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 2676.063829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2676.099791] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2676.106291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2676.111606] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2676.114018] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2676.121159] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2676.123517] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2676.125314] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2676.130113] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2676.132831] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2676.137518] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2676.142897] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2676.148231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2676.151411] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2676.154841] Call Trace: [ 2676.156451] [ 2676.156884] ? show_regs.cold.9+0x22/0x2f [ 2676.157885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2676.159575] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2676.162503] ? watchdog+0x40/0x40 [ 2676.164499] ? hrtimer_interrupt+0x128/0x2c0 [ 2676.167191] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2676.170391] ? apic_timer_interrupt+0xf/0x20 [ 2676.173835] [ 2676.175362] ? native_safe_halt+0x2e/0x50 [ 2676.177426] kvm_wait+0xcd/0xe0 [ 2676.178866] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2676.180547] do_raw_spin_lock+0x177/0x190 [ 2676.181717] _raw_spin_lock+0x21/0x30 [ 2676.182902] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2676.184476] obd_mod_free+0x46/0x140 [ptlrpc] [ 2676.185673] mdc_commit_open+0x91/0xb0 [mdc] [ 2676.187378] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2676.189717] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2676.192454] after_reply+0xe2c/0x1780 [ptlrpc] [ 2676.194707] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2676.197184] ? __next_timer_interrupt+0x160/0x160 [ 2676.199209] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2676.202798] ? do_wait_intr+0xf0/0xf0 [ 2676.205697] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2676.210160] kthread+0x1d1/0x200 [ 2676.211966] ? set_kthread_struct+0x70/0x70 [ 2676.214221] ret_from_fork+0x1f/0x30 [ 2692.053025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 2692.055958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [ 2692.056048] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 2692.056051] osc(O) [ 2692.056057] Modules linked in: [ 2692.066750] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 2692.069522] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2692.069747] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2692.069753] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2692.069755] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2692.069786] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2692.069789] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2692.069792] RAX: 0000000000006f79 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2692.069794] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2692.069796] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2692.069797] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2692.069799] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2692.069801] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2692.069803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2692.069805] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2692.069811] Call Trace: [ 2692.069820] [ 2692.069835] ? show_regs.cold.9+0x22/0x2f [ 2692.069843] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2692.069848] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2692.069850] ? watchdog+0x40/0x40 [ 2692.069853] ? hrtimer_interrupt+0x128/0x2c0 [ 2692.069855] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2692.069861] ? apic_timer_interrupt+0xf/0x20 [ 2692.069865] [ 2692.069867] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2692.069871] do_raw_spin_lock+0x177/0x190 [ 2692.069874] _raw_spin_lock+0x21/0x30 [ 2692.069877] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2692.070126] ? do_raw_spin_unlock+0x75/0x190 [ 2692.070132] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2692.070326] ? do_wait_intr+0xf0/0xf0 [ 2692.070333] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2692.070527] mdc_close+0x53b/0xd70 [mdc] [ 2692.070563] lmv_close+0x18d/0x490 [lmv] [ 2692.070593] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2692.070953] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2692.071032] ll_md_close+0x274/0x800 [lustre] [ 2692.071115] ll_file_release+0x128/0xc50 [lustre] [ 2692.071197] __fput+0x10d/0x440 [ 2692.071207] ____fput+0x12/0x20 [ 2692.071211] task_work_run+0xc6/0x110 [ 2692.071218] exit_to_usermode_loop+0x1dd/0x1f0 [ 2692.071225] do_syscall_64+0x3d6/0x440 [ 2692.071229] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2692.071235] RIP: 0033:0x7f1a63983cd8 [ 2692.071241] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2692.071244] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2692.071248] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2692.071251] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2692.071253] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2692.071255] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2692.071257] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2692.271090] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2692.292654] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2692.300252] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2692.304403] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2692.307398] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2692.317559] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2692.320653] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2692.322708] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2692.328120] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2692.331799] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2692.336077] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2692.339966] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2692.343803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2692.346204] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2692.350130] Call Trace: [ 2692.351241] [ 2692.352154] ? show_regs.cold.9+0x22/0x2f [ 2692.354304] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2692.355750] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2692.358090] ? watchdog+0x40/0x40 [ 2692.359310] ? hrtimer_interrupt+0x128/0x2c0 [ 2692.361076] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2692.363706] ? apic_timer_interrupt+0xf/0x20 [ 2692.365728] [ 2692.366548] ? native_safe_halt+0x2e/0x50 [ 2692.368252] kvm_wait+0xcd/0xe0 [ 2692.369146] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2692.371213] do_raw_spin_lock+0x177/0x190 [ 2692.372736] _raw_spin_lock+0x21/0x30 [ 2692.374700] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2692.377709] ? do_raw_spin_unlock+0x75/0x190 [ 2692.379575] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2692.381967] ? do_wait_intr+0xf0/0xf0 [ 2692.383307] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2692.385789] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2692.388298] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2692.391223] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2692.393610] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2692.396359] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2692.398527] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2692.400896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2692.403250] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2692.406510] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2692.408706] lmv_intent_open+0x338/0x1150 [lmv] [ 2692.410637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2692.413261] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2692.416180] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2692.419896] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2692.422112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2692.425771] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2692.428585] ? do_raw_spin_unlock+0x75/0x190 [ 2692.430638] ? _raw_spin_unlock+0x12/0x30 [ 2692.432445] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2692.435180] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2692.437419] ll_atomic_open+0x36a/0xd70 [lustre] [ 2692.439451] lookup_open+0x356/0xb40 [ 2692.441058] do_last+0x313/0xfc0 [ 2692.442383] ? nd_jump_root+0xe5/0x160 [ 2692.443971] ? path_init+0x437/0x520 [ 2692.445668] path_openat+0xf7/0x500 [ 2692.446851] ? do_raw_spin_unlock+0x75/0x190 [ 2692.448084] do_filp_open+0x99/0x140 [ 2692.449250] ? getname_flags+0x6e/0x330 [ 2692.451080] ? __check_object_size+0xff/0x256 [ 2692.453541] ? do_raw_spin_unlock+0x75/0x190 [ 2692.456510] ? _raw_spin_unlock+0x12/0x30 [ 2692.458580] do_sys_openat2+0x2b4/0x410 [ 2692.460387] do_sys_open+0x73/0xa0 [ 2692.462217] __x64_sys_openat+0x24/0x30 [ 2692.463996] do_syscall_64+0xc1/0x440 [ 2692.465440] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2692.467497] RIP: 0033:0x7f1a63983332 [ 2692.469578] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2692.475316] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2692.478211] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2692.480402] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2692.482670] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2692.485461] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2692.487946] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2696.057031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2696.064317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2696.086943] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2696.091973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2696.096075] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2696.098204] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2696.106518] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2696.110232] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2696.113211] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2696.116308] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2696.128611] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2696.134207] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2696.140276] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2696.144089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2696.146602] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2696.149668] Call Trace: [ 2696.150559] [ 2696.151224] ? show_regs.cold.9+0x22/0x2f [ 2696.152559] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2696.154538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2696.156090] ? watchdog+0x40/0x40 [ 2696.156986] ? hrtimer_interrupt+0x128/0x2c0 [ 2696.158662] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2696.161021] ? apic_timer_interrupt+0xf/0x20 [ 2696.162400] [ 2696.163482] ? native_safe_halt+0x2e/0x50 [ 2696.165236] kvm_wait+0xcd/0xe0 [ 2696.166488] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2696.169054] do_raw_spin_lock+0x177/0x190 [ 2696.170696] _raw_spin_lock+0x21/0x30 [ 2696.172953] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2696.175062] ? do_raw_spin_unlock+0x75/0x190 [ 2696.177661] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2696.180298] ? do_wait_intr+0xf0/0xf0 [ 2696.181384] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2696.188293] mdc_close+0x53b/0xd70 [mdc] [ 2696.189575] lmv_close+0x18d/0x490 [lmv] [ 2696.190766] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2696.193353] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2696.195239] ll_md_close+0x274/0x800 [lustre] [ 2696.196567] ll_file_release+0x128/0xc50 [lustre] [ 2696.198379] __fput+0x10d/0x440 [ 2696.199348] ____fput+0x12/0x20 [ 2696.200351] task_work_run+0xc6/0x110 [ 2696.201735] exit_to_usermode_loop+0x1dd/0x1f0 [ 2696.203200] do_syscall_64+0x3d6/0x440 [ 2696.204589] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2696.206379] RIP: 0033:0x7f1a63983cd8 [ 2696.207497] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2696.212601] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2696.214397] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2696.216177] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2696.218110] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2696.219991] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2696.221570] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2697.696092] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 674s! [ 2697.708274] Showing busy workqueues and worker pools: [ 2697.715436] workqueue events: flags=0x0 [ 2697.716965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2697.730234] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2697.735115] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2697.745229] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2697.756173] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2697.763455] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2697.775337] workqueue events_long: flags=0x0 [ 2697.780184] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2697.785498] pending: lu_objects_flush [obdclass] [ 2697.794245] workqueue events_unbound: flags=0x2 [ 2697.795970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2697.802931] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2697.814990] workqueue events_power_efficient: flags=0x80 [ 2697.820072] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2697.833982] pending: neigh_periodic_work, do_cache_clean [ 2697.841196] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2697.853162] pending: neigh_periodic_work, check_lifetime [ 2697.863355] workqueue mm_percpu_wq: flags=0x8 [ 2697.871452] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2697.884371] pending: vmstat_update [ 2697.885840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2697.898982] pending: vmstat_update [ 2697.906305] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2697.912719] pending: vmstat_update [ 2697.917250] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2697.926408] pending: vmstat_update [ 2697.932545] workqueue writeback: flags=0x4a [ 2697.936117] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2697.940750] pending: wb_workfn, wb_workfn [ 2697.942722] workqueue xprtiod: flags=0xa [ 2697.944962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2697.949627] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2697.954022] workqueue ipv6_addrconf: flags=0x40008 [ 2697.957416] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2697.963425] pending: addrconf_verify_work [ 2697.966633] workqueue knbd0-recv: flags=0x1a [ 2697.970932] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2697.982869] in-flight: 647:recv_work [ 2697.984428] workqueue ptlrpc_pinger: flags=0x2 [ 2697.990474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2697.999359] pending: ptlrpc_pinger_main [ptlrpc] [ 2698.007424] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2688s workers=2 idle: 652 [ 2704.062766] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 2704.071229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2704.137591] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2704.154412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2704.161340] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2704.167646] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2704.192450] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2704.202904] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2704.205601] RDX: 0000000000007ad1 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2704.217725] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2704.223388] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2704.230557] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2704.239268] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2704.247356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2704.255816] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2704.271340] Call Trace: [ 2704.274844] [ 2704.276482] ? show_regs.cold.9+0x22/0x2f [ 2704.279642] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2704.283472] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2704.286334] ? watchdog+0x40/0x40 [ 2704.290544] ? hrtimer_interrupt+0x128/0x2c0 [ 2704.295850] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2704.300244] ? apic_timer_interrupt+0xf/0x20 [ 2704.306182] [ 2704.307762] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2704.317203] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2704.323349] do_raw_spin_lock+0x177/0x190 [ 2704.330084] _raw_spin_lock+0x21/0x30 [ 2704.333443] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2704.345360] obd_mod_free+0x46/0x140 [ptlrpc] [ 2704.347557] mdc_commit_open+0x91/0xb0 [mdc] [ 2704.356350] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2704.358475] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2704.368543] after_reply+0xe2c/0x1780 [ptlrpc] [ 2704.375588] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2704.378550] ? __next_timer_interrupt+0x160/0x160 [ 2704.387294] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2704.391920] ? do_wait_intr+0xf0/0xf0 [ 2704.399031] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2704.405630] kthread+0x1d1/0x200 [ 2704.406927] ? set_kthread_struct+0x70/0x70 [ 2704.410523] ret_from_fork+0x1f/0x30 [ 2720.056024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 2720.060113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2720.079111] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2720.083697] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2720.088873] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2720.090133] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2720.096321] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2720.099114] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2720.101854] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2720.104357] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2720.107096] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2720.109777] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2720.112848] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2720.116627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2720.119531] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2720.124104] Call Trace: [ 2720.125092] [ 2720.125769] ? show_regs.cold.9+0x22/0x2f [ 2720.127432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2720.129422] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2720.131602] ? watchdog+0x40/0x40 [ 2720.132735] ? hrtimer_interrupt+0x128/0x2c0 [ 2720.134369] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2720.136275] ? apic_timer_interrupt+0xf/0x20 [ 2720.137839] [ 2720.138669] ? native_safe_halt+0x2e/0x50 [ 2720.140075] kvm_wait+0xcd/0xe0 [ 2720.141174] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2720.143070] do_raw_spin_lock+0x177/0x190 [ 2720.144432] _raw_spin_lock+0x21/0x30 [ 2720.146405] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2720.149820] ? do_raw_spin_unlock+0x75/0x190 [ 2720.151971] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2720.154001] ? do_wait_intr+0xf0/0xf0 [ 2720.155350] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2720.157490] mdc_close+0x53b/0xd70 [mdc] [ 2720.158866] lmv_close+0x18d/0x490 [lmv] [ 2720.160157] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2720.162261] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2720.163778] ll_md_close+0x274/0x800 [lustre] [ 2720.165395] ll_file_release+0x128/0xc50 [lustre] [ 2720.167268] __fput+0x10d/0x440 [ 2720.168417] ____fput+0x12/0x20 [ 2720.169709] task_work_run+0xc6/0x110 [ 2720.170887] exit_to_usermode_loop+0x1dd/0x1f0 [ 2720.172726] do_syscall_64+0x3d6/0x440 [ 2720.173950] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2720.176257] RIP: 0033:0x7f1a63983cd8 [ 2720.177433] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2720.186366] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2720.189916] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2720.193211] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2720.196269] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2720.200073] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2720.203519] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2724.053063] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2724.057922] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2724.059078] Modules linked in: lustre(O) [ 2724.062812] Modules linked in: lustre(O) osp(O) [ 2724.065447] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [ 2724.069735] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2724.069806] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2724.069812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2724.069815] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2724.069828] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2724.075476] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2724.075556] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2724.075563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2724.075566] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2724.075741] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2724.075746] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2724.075750] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2724.075753] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2724.075755] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2724.075757] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2724.075759] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2724.075899] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2724.075902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2724.075904] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2724.075912] Call Trace: [ 2724.075917] [ 2724.075931] ? show_regs.cold.9+0x22/0x2f [ 2724.076066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2724.076073] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2724.076077] ? watchdog+0x40/0x40 [ 2724.076081] ? hrtimer_interrupt+0x128/0x2c0 [ 2724.076085] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2724.076092] ? apic_timer_interrupt+0xf/0x20 [ 2724.076096] [ 2724.076099] ? native_safe_halt+0x2e/0x50 [ 2724.076103] kvm_wait+0xcd/0xe0 [ 2724.076110] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2724.076117] do_raw_spin_lock+0x177/0x190 [ 2724.076122] _raw_spin_lock+0x21/0x30 [ 2724.076126] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2724.099293] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2724.099303] RAX: 0000000000005210 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2724.099306] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2724.099308] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2724.099310] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2724.099312] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2724.099315] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2724.099318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2724.099320] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2724.099330] Call Trace: [ 2724.099336] [ 2724.108212] ? do_raw_spin_unlock+0x75/0x190 [ 2724.115942] ? show_regs.cold.9+0x22/0x2f [ 2724.121341] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2724.133431] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2724.133601] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2724.133606] ? watchdog+0x40/0x40 [ 2724.133610] ? hrtimer_interrupt+0x128/0x2c0 [ 2724.133613] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2724.133620] ? apic_timer_interrupt+0xf/0x20 [ 2724.133624] [ 2724.133629] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2724.157146] ? do_wait_intr+0xf0/0xf0 [ 2724.157169] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2724.157660] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2724.157877] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2724.158179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2724.158308] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2724.158348] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2724.158602] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2724.158629] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2724.158718] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2724.158952] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2724.158980] lmv_intent_open+0x338/0x1150 [lmv] [ 2724.159007] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2724.159102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2724.159189] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2724.159211] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2724.159362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2724.159660] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2724.159745] ? do_raw_spin_unlock+0x75/0x190 [ 2724.159753] ? _raw_spin_unlock+0x12/0x30 [ 2724.159759] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2724.159790] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2724.159810] ll_atomic_open+0x36a/0xd70 [lustre] [ 2724.159900] lookup_open+0x356/0xb40 [ 2724.159909] do_last+0x313/0xfc0 [ 2724.159915] ? nd_jump_root+0xe5/0x160 [ 2724.159920] ? path_init+0x437/0x520 [ 2724.159924] path_openat+0xf7/0x500 [ 2724.159929] ? do_raw_spin_unlock+0x75/0x190 [ 2724.159934] do_filp_open+0x99/0x140 [ 2724.159941] ? getname_flags+0x6e/0x330 [ 2724.159945] ? __check_object_size+0xff/0x256 [ 2724.159953] ? do_raw_spin_unlock+0x75/0x190 [ 2724.159957] ? _raw_spin_unlock+0x12/0x30 [ 2724.159962] do_sys_openat2+0x2b4/0x410 [ 2724.159967] do_sys_open+0x73/0xa0 [ 2724.159970] __x64_sys_openat+0x24/0x30 [ 2724.159974] do_syscall_64+0xc1/0x440 [ 2724.159981] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2724.159987] RIP: 0033:0x7f1a63983332 [ 2724.159995] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2724.159999] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2724.160005] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2724.160007] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2724.160010] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2724.160011] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2724.160013] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2724.442483] do_raw_spin_lock+0x177/0x190 [ 2724.444654] _raw_spin_lock+0x21/0x30 [ 2724.447239] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2724.449591] ? do_raw_spin_unlock+0x75/0x190 [ 2724.451319] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2724.453800] ? do_wait_intr+0xf0/0xf0 [ 2724.455443] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2724.458366] mdc_close+0x53b/0xd70 [mdc] [ 2724.460286] lmv_close+0x18d/0x490 [lmv] [ 2724.462052] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2724.464801] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2724.467250] ll_md_close+0x274/0x800 [lustre] [ 2724.468967] ll_file_release+0x128/0xc50 [lustre] [ 2724.471579] __fput+0x10d/0x440 [ 2724.472983] ____fput+0x12/0x20 [ 2724.474293] task_work_run+0xc6/0x110 [ 2724.476101] exit_to_usermode_loop+0x1dd/0x1f0 [ 2724.478052] do_syscall_64+0x3d6/0x440 [ 2724.479343] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2724.482123] RIP: 0033:0x7f1a63983cd8 [ 2724.484318] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2724.492456] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2724.495616] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2724.499097] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2724.502445] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2724.505517] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2724.508809] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2728.419591] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 705s! [ 2728.428191] Showing busy workqueues and worker pools: [ 2728.431196] workqueue events: flags=0x0 [ 2728.433075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.439863] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2728.449752] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2728.455998] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2728.471962] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2728.479227] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2728.491163] workqueue events_long: flags=0x0 [ 2728.494022] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2728.498034] pending: lu_objects_flush [obdclass] [ 2728.501747] workqueue events_unbound: flags=0x2 [ 2728.504559] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2728.508888] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2728.513626] workqueue events_power_efficient: flags=0x80 [ 2728.516224] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.519277] pending: neigh_periodic_work, do_cache_clean [ 2728.521621] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.524156] pending: neigh_periodic_work, check_lifetime [ 2728.526226] workqueue mm_percpu_wq: flags=0x8 [ 2728.527959] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2728.530416] pending: vmstat_update [ 2728.532198] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2728.534817] pending: vmstat_update [ 2728.536263] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2728.538759] pending: vmstat_update [ 2728.540150] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2728.542564] pending: vmstat_update [ 2728.544066] workqueue writeback: flags=0x4a [ 2728.545527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2728.547943] pending: wb_workfn, wb_workfn [ 2728.549442] workqueue xprtiod: flags=0xa [ 2728.551234] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2728.553402] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2728.557034] workqueue ipv6_addrconf: flags=0x40008 [ 2728.559193] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2728.561938] pending: addrconf_verify_work [ 2728.563925] workqueue knbd0-recv: flags=0x1a [ 2728.565681] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2728.568043] in-flight: 647:recv_work [ 2728.569867] workqueue ptlrpc_pinger: flags=0x2 [ 2728.571966] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2728.574797] pending: ptlrpc_pinger_main [ptlrpc] [ 2728.577307] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2718s workers=2 idle: 652 [ 2732.060000] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2732.065410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2732.095673] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2732.105387] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2732.112326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2732.124324] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2732.141297] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2732.144121] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2732.147967] RDX: 0000000000001f08 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2732.154380] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2732.171990] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2732.187691] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2732.198798] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2732.203094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2732.207177] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2732.211851] Call Trace: [ 2732.214945] [ 2732.216164] ? show_regs.cold.9+0x22/0x2f [ 2732.218020] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2732.221051] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2732.223975] ? watchdog+0x40/0x40 [ 2732.225548] ? hrtimer_interrupt+0x128/0x2c0 [ 2732.227418] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2732.230070] ? apic_timer_interrupt+0xf/0x20 [ 2732.232008] [ 2732.232852] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2732.235017] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2732.239172] do_raw_spin_lock+0x177/0x190 [ 2732.241028] _raw_spin_lock+0x21/0x30 [ 2732.242831] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2732.245856] obd_mod_free+0x46/0x140 [ptlrpc] [ 2732.248667] mdc_commit_open+0x91/0xb0 [mdc] [ 2732.253228] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2732.255326] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2732.260063] after_reply+0xe2c/0x1780 [ptlrpc] [ 2732.262547] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2732.265104] ? __next_timer_interrupt+0x160/0x160 [ 2732.268928] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2732.276509] ? do_wait_intr+0xf0/0xf0 [ 2732.277794] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2732.283007] kthread+0x1d1/0x200 [ 2732.285643] ? set_kthread_struct+0x70/0x70 [ 2732.288188] ret_from_fork+0x1f/0x30 [ 2748.056772] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 2748.059654] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2748.092183] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2748.097409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2748.101218] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2748.103361] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2748.112009] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2748.115487] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2748.118779] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2748.122296] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2748.125654] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2748.129181] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2748.132618] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2748.136385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2748.138969] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2748.142457] Call Trace: [ 2748.143658] [ 2748.144713] ? show_regs.cold.9+0x22/0x2f [ 2748.146618] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2748.148659] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2748.150756] ? watchdog+0x40/0x40 [ 2748.152169] ? hrtimer_interrupt+0x128/0x2c0 [ 2748.153957] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2748.156018] ? apic_timer_interrupt+0xf/0x20 [ 2748.157936] [ 2748.158798] ? native_safe_halt+0x2e/0x50 [ 2748.160708] kvm_wait+0xcd/0xe0 [ 2748.162054] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2748.164443] do_raw_spin_lock+0x177/0x190 [ 2748.166165] _raw_spin_lock+0x21/0x30 [ 2748.168348] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2748.171622] ? do_raw_spin_unlock+0x75/0x190 [ 2748.173611] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2748.175974] ? do_wait_intr+0xf0/0xf0 [ 2748.177607] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2748.180161] mdc_close+0x53b/0xd70 [mdc] [ 2748.181946] lmv_close+0x18d/0x490 [lmv] [ 2748.183653] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2748.186401] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2748.188538] ll_md_close+0x274/0x800 [lustre] [ 2748.190822] ll_file_release+0x128/0xc50 [lustre] [ 2748.195866] __fput+0x10d/0x440 [ 2748.197265] ____fput+0x12/0x20 [ 2748.199065] task_work_run+0xc6/0x110 [ 2748.200584] exit_to_usermode_loop+0x1dd/0x1f0 [ 2748.202648] do_syscall_64+0x3d6/0x440 [ 2748.204503] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2748.206859] RIP: 0033:0x7f1a63983cd8 [ 2748.208588] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2748.217513] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2748.221169] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2748.224915] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2748.228618] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2748.232363] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2748.235819] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2752.057022] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 2752.065173] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2752.108816] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2752.116684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2752.122571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2752.127789] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2752.143704] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2752.153114] RAX: 0000000000006167 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2752.157978] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2752.166329] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2752.175969] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2752.180139] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2752.185597] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2752.192313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2752.198637] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2752.208735] Call Trace: [ 2752.209969] [ 2752.210842] ? show_regs.cold.9+0x22/0x2f [ 2752.216922] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2752.223995] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2752.228962] ? watchdog+0x40/0x40 [ 2752.232908] ? hrtimer_interrupt+0x128/0x2c0 [ 2752.239912] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2752.248448] ? apic_timer_interrupt+0xf/0x20 [ 2752.250761] [ 2752.253893] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2752.261232] do_raw_spin_lock+0x177/0x190 [ 2752.280435] _raw_spin_lock+0x21/0x30 [ 2752.288143] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2752.298260] ? do_raw_spin_unlock+0x75/0x190 [ 2752.300845] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2752.307603] ? do_wait_intr+0xf0/0xf0 [ 2752.312417] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2752.318468] mdc_close+0x53b/0xd70 [mdc] [ 2752.322555] lmv_close+0x18d/0x490 [lmv] [ 2752.328509] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2752.336144] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2752.343736] ll_md_close+0x274/0x800 [lustre] [ 2752.347275] ll_file_release+0x128/0xc50 [lustre] [ 2752.350722] __fput+0x10d/0x440 [ 2752.351915] ____fput+0x12/0x20 [ 2752.353042] task_work_run+0xc6/0x110 [ 2752.356551] exit_to_usermode_loop+0x1dd/0x1f0 [ 2752.359191] do_syscall_64+0x3d6/0x440 [ 2752.362179] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2752.365609] RIP: 0033:0x7f1a63983cd8 [ 2752.366858] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2752.378774] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2752.385572] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2752.393978] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2752.403466] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2752.410580] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2752.419791] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2756.053036] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2756.056026] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2756.098089] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2756.103745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2756.113147] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2756.114754] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2756.120947] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2756.122900] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2756.124991] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2756.127820] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2756.130633] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2756.133355] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2756.136016] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2756.138629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2756.140321] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2756.142849] Call Trace: [ 2756.146964] [ 2756.147739] ? show_regs.cold.9+0x22/0x2f [ 2756.149330] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2756.152016] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2756.153990] ? watchdog+0x40/0x40 [ 2756.155351] ? hrtimer_interrupt+0x128/0x2c0 [ 2756.157032] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2756.158948] ? apic_timer_interrupt+0xf/0x20 [ 2756.160724] [ 2756.161558] ? native_safe_halt+0x2e/0x50 [ 2756.165164] kvm_wait+0xcd/0xe0 [ 2756.166310] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2756.174781] do_raw_spin_lock+0x177/0x190 [ 2756.179151] _raw_spin_lock+0x21/0x30 [ 2756.180414] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2756.183192] ? do_raw_spin_unlock+0x75/0x190 [ 2756.184568] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2756.188222] ? do_wait_intr+0xf0/0xf0 [ 2756.189766] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2756.192000] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2756.194157] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2756.196771] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2756.199115] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2756.201246] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2756.203036] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2756.204678] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2756.206613] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2756.209465] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2756.211108] lmv_intent_open+0x338/0x1150 [lmv] [ 2756.212682] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2756.214745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2756.216792] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2756.218642] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2756.220496] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2756.222451] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2756.224001] ? do_raw_spin_unlock+0x75/0x190 [ 2756.225650] ? _raw_spin_unlock+0x12/0x30 [ 2756.227157] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2756.229127] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2756.230891] ll_atomic_open+0x36a/0xd70 [lustre] [ 2756.232659] lookup_open+0x356/0xb40 [ 2756.234074] do_last+0x313/0xfc0 [ 2756.235323] ? nd_jump_root+0xe5/0x160 [ 2756.236719] ? path_init+0x437/0x520 [ 2756.237865] path_openat+0xf7/0x500 [ 2756.239032] ? do_raw_spin_unlock+0x75/0x190 [ 2756.240627] do_filp_open+0x99/0x140 [ 2756.241733] ? getname_flags+0x6e/0x330 [ 2756.242839] ? __check_object_size+0xff/0x256 [ 2756.244353] ? do_raw_spin_unlock+0x75/0x190 [ 2756.245802] ? _raw_spin_unlock+0x12/0x30 [ 2756.247372] do_sys_openat2+0x2b4/0x410 [ 2756.248747] do_sys_open+0x73/0xa0 [ 2756.250047] __x64_sys_openat+0x24/0x30 [ 2756.251493] do_syscall_64+0xc1/0x440 [ 2756.252722] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2756.254444] RIP: 0033:0x7f1a63983332 [ 2756.255551] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2756.261874] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2756.264703] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2756.267439] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2756.270034] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2756.272653] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2756.275392] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2758.629039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 735s! [ 2758.641112] Showing busy workqueues and worker pools: [ 2758.661715] workqueue events: flags=0x0 [ 2758.675488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2758.692402] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2758.701558] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2758.710337] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2758.718535] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2758.726393] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2758.733942] workqueue events_long: flags=0x0 [ 2758.744486] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2758.753802] pending: lu_objects_flush [obdclass] [ 2758.760446] workqueue events_unbound: flags=0x2 [ 2758.763808] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2758.770474] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2758.780084] workqueue events_power_efficient: flags=0x80 [ 2758.797915] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2758.802656] pending: neigh_periodic_work, do_cache_clean [ 2758.808585] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2758.815988] pending: neigh_periodic_work, check_lifetime [ 2758.821459] workqueue mm_percpu_wq: flags=0x8 [ 2758.825691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2758.833056] pending: vmstat_update [ 2758.842552] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2758.858022] pending: vmstat_update [ 2758.859227] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2758.862610] pending: vmstat_update [ 2758.866705] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2758.871319] pending: vmstat_update [ 2758.878545] workqueue writeback: flags=0x4a [ 2758.884658] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2758.890790] pending: wb_workfn, wb_workfn [ 2758.898953] workqueue xprtiod: flags=0xa [ 2758.900453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2758.910165] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2758.913156] workqueue ipv6_addrconf: flags=0x40008 [ 2758.919876] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2758.930044] pending: addrconf_verify_work [ 2758.938163] workqueue knbd0-recv: flags=0x1a [ 2758.939746] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2758.955611] in-flight: 647:recv_work [ 2758.961687] workqueue ptlrpc_pinger: flags=0x2 [ 2758.969588] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2758.975531] pending: ptlrpc_pinger_main [ptlrpc] [ 2758.990521] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2749s workers=2 idle: 652 [ 2760.059112] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2760.062564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2760.088478] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2760.093231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2760.097145] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2760.099934] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2760.108824] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2760.113039] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2760.116085] RDX: 0000000000007687 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2760.119450] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2760.122577] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2760.125642] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2760.128874] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2760.132787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2760.135747] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2760.138825] Call Trace: [ 2760.140026] [ 2760.140962] ? show_regs.cold.9+0x22/0x2f [ 2760.143111] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2760.145602] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2760.147263] ? watchdog+0x40/0x40 [ 2760.148620] ? hrtimer_interrupt+0x128/0x2c0 [ 2760.150698] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2760.152806] ? apic_timer_interrupt+0xf/0x20 [ 2760.155445] [ 2760.156512] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2760.159069] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2760.161833] do_raw_spin_lock+0x177/0x190 [ 2760.163854] _raw_spin_lock+0x21/0x30 [ 2760.165534] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2760.168354] obd_mod_free+0x46/0x140 [ptlrpc] [ 2760.171047] mdc_commit_open+0x91/0xb0 [mdc] [ 2760.173049] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2760.175727] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2760.178663] after_reply+0xe2c/0x1780 [ptlrpc] [ 2760.181592] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2760.184186] ? __next_timer_interrupt+0x160/0x160 [ 2760.186515] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2760.189241] ? do_wait_intr+0xf0/0xf0 [ 2760.190752] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2760.193464] kthread+0x1d1/0x200 [ 2760.194852] ? set_kthread_struct+0x70/0x70 [ 2760.197170] ret_from_fork+0x1f/0x30 [ 2776.058270] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2776.061703] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2776.082351] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2776.088863] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2776.094332] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2776.097998] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2776.112818] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2776.119776] RAX: 0000000000006486 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2776.123325] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2776.126588] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2776.130470] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2776.133945] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2776.137071] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2776.141355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2776.143670] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2776.147405] Call Trace: [ 2776.149472] [ 2776.150485] ? show_regs.cold.9+0x22/0x2f [ 2776.152485] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2776.155178] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2776.159177] ? watchdog+0x40/0x40 [ 2776.160710] ? hrtimer_interrupt+0x128/0x2c0 [ 2776.163151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2776.165148] ? apic_timer_interrupt+0xf/0x20 [ 2776.167522] [ 2776.168801] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2776.172821] do_raw_spin_lock+0x177/0x190 [ 2776.174746] _raw_spin_lock+0x21/0x30 [ 2776.179378] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2776.187686] ? do_raw_spin_unlock+0x75/0x190 [ 2776.190982] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2776.198296] ? do_wait_intr+0xf0/0xf0 [ 2776.203648] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2776.208670] mdc_close+0x53b/0xd70 [mdc] [ 2776.210460] lmv_close+0x18d/0x490 [lmv] [ 2776.212620] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2776.215333] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2776.217763] ll_md_close+0x274/0x800 [lustre] [ 2776.219709] ll_file_release+0x128/0xc50 [lustre] [ 2776.222744] __fput+0x10d/0x440 [ 2776.224744] ____fput+0x12/0x20 [ 2776.226206] task_work_run+0xc6/0x110 [ 2776.228153] exit_to_usermode_loop+0x1dd/0x1f0 [ 2776.229855] do_syscall_64+0x3d6/0x440 [ 2776.233590] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2776.235658] RIP: 0033:0x7f1a63983cd8 [ 2776.237401] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2776.245281] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2776.250698] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2776.258759] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2776.263017] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2776.266909] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2776.274555] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2780.063376] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 2780.080440] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2780.123703] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2780.137673] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2780.145951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2780.150260] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2780.160456] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2780.166179] RAX: 000000000000781b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2780.172082] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2780.177853] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2780.187911] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2780.195852] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2780.206913] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2780.212645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2780.216417] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2780.225313] Call Trace: [ 2780.227764] [ 2780.230721] ? show_regs.cold.9+0x22/0x2f [ 2780.233708] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2780.241840] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2780.248245] ? watchdog+0x40/0x40 [ 2780.254055] ? hrtimer_interrupt+0x128/0x2c0 [ 2780.256360] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2780.263568] ? apic_timer_interrupt+0xf/0x20 [ 2780.268841] [ 2780.269866] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2780.275648] do_raw_spin_lock+0x177/0x190 [ 2780.279411] _raw_spin_lock+0x21/0x30 [ 2780.284344] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2780.288543] ? do_raw_spin_unlock+0x75/0x190 [ 2780.291704] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2780.297756] ? do_wait_intr+0xf0/0xf0 [ 2780.299736] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2780.302881] mdc_close+0x53b/0xd70 [mdc] [ 2780.305466] lmv_close+0x18d/0x490 [lmv] [ 2780.311729] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2780.313646] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2780.327743] ll_md_close+0x274/0x800 [lustre] [ 2780.330868] ll_file_release+0x128/0xc50 [lustre] [ 2780.337539] __fput+0x10d/0x440 [ 2780.343228] ____fput+0x12/0x20 [ 2780.347565] task_work_run+0xc6/0x110 [ 2780.348783] exit_to_usermode_loop+0x1dd/0x1f0 [ 2780.353082] do_syscall_64+0x3d6/0x440 [ 2780.359523] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2780.363537] RIP: 0033:0x7f1a63983cd8 [ 2780.367710] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2780.384716] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2780.390426] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2780.392795] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2780.397909] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2780.404607] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2780.409268] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2784.053035] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2784.056617] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2784.094870] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2784.101916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2784.105632] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2784.108018] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2784.120221] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2784.125152] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2784.129511] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2784.132825] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2784.136476] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2784.143383] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2784.148914] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2784.154397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2784.158030] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2784.161731] Call Trace: [ 2784.163366] [ 2784.164702] ? show_regs.cold.9+0x22/0x2f [ 2784.168589] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2784.171260] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2784.175266] ? watchdog+0x40/0x40 [ 2784.181023] ? hrtimer_interrupt+0x128/0x2c0 [ 2784.187934] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2784.192066] ? apic_timer_interrupt+0xf/0x20 [ 2784.194716] [ 2784.196043] ? native_safe_halt+0x2e/0x50 [ 2784.198365] kvm_wait+0xcd/0xe0 [ 2784.200617] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2784.204003] do_raw_spin_lock+0x177/0x190 [ 2784.206417] _raw_spin_lock+0x21/0x30 [ 2784.208606] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2784.211909] ? do_raw_spin_unlock+0x75/0x190 [ 2784.214363] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2784.217099] ? do_wait_intr+0xf0/0xf0 [ 2784.219593] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2784.231493] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2784.240380] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2784.244423] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2784.250314] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2784.255077] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2784.258996] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2784.262225] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2784.265826] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2784.271342] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2784.281295] lmv_intent_open+0x338/0x1150 [lmv] [ 2784.290713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2784.299963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2784.308894] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2784.315758] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2784.321575] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2784.325342] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2784.332892] ? do_raw_spin_unlock+0x75/0x190 [ 2784.334548] ? _raw_spin_unlock+0x12/0x30 [ 2784.349043] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2784.357321] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2784.367698] ll_atomic_open+0x36a/0xd70 [lustre] [ 2784.369631] lookup_open+0x356/0xb40 [ 2784.372353] do_last+0x313/0xfc0 [ 2784.374593] ? nd_jump_root+0xe5/0x160 [ 2784.384375] ? path_init+0x437/0x520 [ 2784.394502] path_openat+0xf7/0x500 [ 2784.405027] ? do_raw_spin_unlock+0x75/0x190 [ 2784.419286] do_filp_open+0x99/0x140 [ 2784.420687] ? getname_flags+0x6e/0x330 [ 2784.422081] ? __check_object_size+0xff/0x256 [ 2784.438244] ? do_raw_spin_unlock+0x75/0x190 [ 2784.439756] ? _raw_spin_unlock+0x12/0x30 [ 2784.454475] do_sys_openat2+0x2b4/0x410 [ 2784.457773] do_sys_open+0x73/0xa0 [ 2784.467350] __x64_sys_openat+0x24/0x30 [ 2784.469907] do_syscall_64+0xc1/0x440 [ 2784.480509] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2784.491072] RIP: 0033:0x7f1a63983332 [ 2784.500864] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2784.530487] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2784.541726] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2784.548531] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2784.555056] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2784.560284] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2784.567497] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2788.060282] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2788.065579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2788.094560] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2788.100621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2788.105783] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2788.110063] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2788.120130] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2788.123944] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2788.132758] RDX: 00000000000002d8 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2788.137761] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2788.141827] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2788.145809] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2788.149757] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2788.154573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2788.158572] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2788.162573] Call Trace: [ 2788.165196] [ 2788.169645] ? show_regs.cold.9+0x22/0x2f [ 2788.172377] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2788.174771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2788.176830] ? watchdog+0x40/0x40 [ 2788.178378] ? hrtimer_interrupt+0x128/0x2c0 [ 2788.180346] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2788.182587] ? apic_timer_interrupt+0xf/0x20 [ 2788.183767] [ 2788.184384] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2788.185808] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2788.187369] do_raw_spin_lock+0x177/0x190 [ 2788.188449] _raw_spin_lock+0x21/0x30 [ 2788.189623] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2788.190902] obd_mod_free+0x46/0x140 [ptlrpc] [ 2788.192305] mdc_commit_open+0x91/0xb0 [mdc] [ 2788.193576] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2788.195170] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2788.196854] after_reply+0xe2c/0x1780 [ptlrpc] [ 2788.198341] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2788.199872] ? __next_timer_interrupt+0x160/0x160 [ 2788.201044] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2788.202604] ? do_wait_intr+0xf0/0xf0 [ 2788.203588] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2788.205165] kthread+0x1d1/0x200 [ 2788.206097] ? set_kthread_struct+0x70/0x70 [ 2788.207422] ret_from_fork+0x1f/0x30 [ 2789.347636] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 766s! [ 2789.351767] Showing busy workqueues and worker pools: [ 2789.357487] workqueue events: flags=0x0 [ 2789.365419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2789.379319] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2789.386525] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2789.393808] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2789.412662] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2789.416252] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2789.423784] workqueue events_long: flags=0x0 [ 2789.468569] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2789.475183] pending: lu_objects_flush [obdclass] [ 2789.478374] workqueue events_unbound: flags=0x2 [ 2789.482069] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2789.484091] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2789.487489] workqueue events_power_efficient: flags=0x80 [ 2789.489618] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2789.492172] pending: neigh_periodic_work, do_cache_clean [ 2789.493530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2789.497710] pending: neigh_periodic_work, check_lifetime [ 2789.499160] workqueue mm_percpu_wq: flags=0x8 [ 2789.500126] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2789.501664] pending: vmstat_update [ 2789.503103] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2789.504740] pending: vmstat_update [ 2789.505780] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2789.507375] pending: vmstat_update [ 2789.508571] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2789.510350] pending: vmstat_update [ 2789.511237] workqueue writeback: flags=0x4a [ 2789.512206] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2789.513716] pending: wb_workfn, wb_workfn [ 2789.514870] workqueue xprtiod: flags=0xa [ 2789.515778] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2789.517305] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2789.519390] workqueue ipv6_addrconf: flags=0x40008 [ 2789.520417] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2789.521823] pending: addrconf_verify_work [ 2789.523155] workqueue knbd0-recv: flags=0x1a [ 2789.524270] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2789.526515] in-flight: 647:recv_work [ 2789.527532] workqueue ptlrpc_pinger: flags=0x2 [ 2789.528621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2789.530190] pending: ptlrpc_pinger_main [ptlrpc] [ 2789.531911] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2779s workers=2 idle: 652 [ 2802.435225] rcu: INFO: rcu_sched self-detected stall on CPU [ 2802.448418] rcu: 3-...!: (455568 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 2802.482467] (t=780075 jiffies g=269633 q=441) [ 2802.488352] rcu: rcu_sched kthread starved for 779723 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 2802.507791] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2802.520419] rcu: RCU grace-period kthread stack dump: [ 2802.521709] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2802.526043] Call Trace: [ 2802.528346] __schedule+0x351/0xcb0 [ 2802.530937] schedule+0xc0/0x180 [ 2802.532423] schedule_timeout+0xb4/0x190 [ 2802.534802] ? do_raw_spin_unlock+0x75/0x190 [ 2802.538432] ? __next_timer_interrupt+0x160/0x160 [ 2802.541198] ? prepare_to_swait_event+0xf3/0x140 [ 2802.547286] rcu_gp_kthread+0x8dd/0x15b0 [ 2802.557124] ? force_qs_rnp+0x2c0/0x2c0 [ 2802.561504] kthread+0x1d1/0x200 [ 2802.562390] ? set_kthread_struct+0x70/0x70 [ 2802.571833] ret_from_fork+0x1f/0x30 [ 2802.578268] rcu: Stack dump where RCU GP kthread last ran: [ 2802.586601] Sending NMI from CPU 3 to CPUs 0: [ 2802.601224] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 4.782 msecs [ 2802.604209] NMI backtrace for cpu 0 [ 2802.604216] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2802.604219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2802.604221] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2802.604222] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2802.604224] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 [ 2802.604227] RAX: 0000000000006002 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2802.604229] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2802.604230] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2802.604231] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2802.604233] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2802.604235] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2802.604236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2802.604237] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2802.604239] Call Trace: [ 2802.604240] [ 2802.604240] ? show_regs.cold.9+0x22/0x2f [ 2802.604241] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2802.604242] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2802.604243] ? nmi_handle+0xbd/0x210 [ 2802.604244] ? default_do_nmi+0x4b/0x1b0 [ 2802.604245] ? do_nmi+0x280/0x380 [ 2802.604246] ? end_repeat_nmi+0x16/0x1f [ 2802.604247] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2802.604249] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2802.604250] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2802.604251] [ 2802.604252] do_raw_spin_lock+0x177/0x190 [ 2802.604253] _raw_spin_lock+0x21/0x30 [ 2802.604254] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2802.604256] ? do_raw_spin_unlock+0x75/0x190 [ 2802.604257] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2802.604258] ? do_wait_intr+0xf0/0xf0 [ 2802.604259] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2802.604260] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2802.604262] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2802.604263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2802.604264] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2802.604265] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2802.604266] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2802.604267] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2802.604268] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2802.604269] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2802.604270] lmv_intent_open+0x338/0x1150 [lmv] [ 2802.604272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2802.604273] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2802.604274] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2802.604275] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2802.604276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2802.604277] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2802.604278] ? do_raw_spin_unlock+0x75/0x190 [ 2802.604279] ? _raw_spin_unlock+0x12/0x30 [ 2802.604280] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2802.604281] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2802.604282] ll_atomic_open+0x36a/0xd70 [lustre] [ 2802.604282] lookup_open+0x356/0xb40 [ 2802.604283] do_last+0x313/0xfc0 [ 2802.604284] ? nd_jump_root+0xe5/0x160 [ 2802.604285] ? path_init+0x437/0x520 [ 2802.604286] path_openat+0xf7/0x500 [ 2802.604287] ? do_raw_spin_unlock+0x75/0x190 [ 2802.604287] do_filp_open+0x99/0x140 [ 2802.604288] ? getname_flags+0x6e/0x330 [ 2802.604289] ? __check_object_size+0xff/0x256 [ 2802.604290] ? do_raw_spin_unlock+0x75/0x190 [ 2802.604292] ? _raw_spin_unlock+0x12/0x30 [ 2802.604293] do_sys_openat2+0x2b4/0x410 [ 2802.604294] do_sys_open+0x73/0xa0 [ 2802.604295] __x64_sys_openat+0x24/0x30 [ 2802.604296] do_syscall_64+0xc1/0x440 [ 2802.604297] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2802.604298] RIP: 0033:0x7f1a63983332 [ 2802.604300] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2802.604302] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2802.604304] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2802.604305] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2802.604307] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2802.604308] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2802.604309] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2802.604315] NMI backtrace for cpu 3 [ 2802.955158] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2802.970359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2802.974261] Call Trace: [ 2802.975516] [ 2802.976535] ? dump_stack+0xbb/0x10e [ 2802.980321] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2802.984161] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2802.987674] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2802.993609] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2802.997864] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2803.004634] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2803.007634] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2803.018483] ? update_process_times+0x6d/0xc0 [ 2803.024337] ? tick_sched_handle.isra.18+0x27/0x90 [ 2803.033228] ? tick_sched_timer+0x53/0xb0 [ 2803.034728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2803.037985] ? tick_sched_handle.isra.18+0x90/0x90 [ 2803.039327] ? hrtimer_interrupt+0x128/0x2c0 [ 2803.040462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2803.049693] ? apic_timer_interrupt+0xf/0x20 [ 2803.051525] [ 2803.053066] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2803.058569] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2803.064179] ? do_raw_spin_lock+0x177/0x190 [ 2803.068465] ? _raw_spin_lock+0x21/0x30 [ 2803.071618] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2803.074226] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2803.080231] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2803.085619] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2803.089765] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2803.098311] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2803.102023] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2803.108723] ? __next_timer_interrupt+0x160/0x160 [ 2803.111100] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2803.115583] ? do_wait_intr+0xf0/0xf0 [ 2803.117412] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2803.120569] ? kthread+0x1d1/0x200 [ 2803.122479] ? set_kthread_struct+0x70/0x70 [ 2803.124409] ? ret_from_fork+0x1f/0x30 [ 2804.059470] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2804.068012] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2804.123239] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2804.139177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2804.149898] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2804.159333] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2804.183364] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2804.190319] RAX: 0000000000007b88 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2804.199072] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2804.205160] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2804.213118] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2804.220328] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2804.228082] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2804.236030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2804.246386] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2804.255128] Call Trace: [ 2804.263920] [ 2804.267082] ? show_regs.cold.9+0x22/0x2f [ 2804.275091] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2804.290960] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2804.299103] ? watchdog+0x40/0x40 [ 2804.308535] ? hrtimer_interrupt+0x128/0x2c0 [ 2804.318178] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2804.323094] ? apic_timer_interrupt+0xf/0x20 [ 2804.328713] [ 2804.331047] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2804.334696] do_raw_spin_lock+0x177/0x190 [ 2804.337372] _raw_spin_lock+0x21/0x30 [ 2804.340750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2804.344566] ? do_raw_spin_unlock+0x75/0x190 [ 2804.352366] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2804.358099] ? do_wait_intr+0xf0/0xf0 [ 2804.361922] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2804.366692] mdc_close+0x53b/0xd70 [mdc] [ 2804.369495] lmv_close+0x18d/0x490 [lmv] [ 2804.376459] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2804.381352] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2804.388349] ll_md_close+0x274/0x800 [lustre] [ 2804.394957] ll_file_release+0x128/0xc50 [lustre] [ 2804.402431] __fput+0x10d/0x440 [ 2804.405005] ____fput+0x12/0x20 [ 2804.406419] task_work_run+0xc6/0x110 [ 2804.416985] exit_to_usermode_loop+0x1dd/0x1f0 [ 2804.420380] do_syscall_64+0x3d6/0x440 [ 2804.423257] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2804.426153] RIP: 0033:0x7f1a63983cd8 [ 2804.428141] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2804.441722] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2804.445399] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2804.456388] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2804.462362] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2804.466883] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2804.470342] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2808.059781] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 2808.066018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2808.124538] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2808.148246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2808.180120] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2808.202739] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2808.217586] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2808.229302] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2808.235967] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2808.248432] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2808.262187] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2808.269718] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2808.281764] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2808.293339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2808.303667] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2808.316670] Call Trace: [ 2808.325008] [ 2808.329008] ? show_regs.cold.9+0x22/0x2f [ 2808.334273] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2808.345944] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2808.358291] ? watchdog+0x40/0x40 [ 2808.364742] ? hrtimer_interrupt+0x128/0x2c0 [ 2808.368057] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2808.373950] ? apic_timer_interrupt+0xf/0x20 [ 2808.381679] [ 2808.385428] ? native_safe_halt+0x2e/0x50 [ 2808.387487] kvm_wait+0xcd/0xe0 [ 2808.388616] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2808.396286] do_raw_spin_lock+0x177/0x190 [ 2808.399625] _raw_spin_lock+0x21/0x30 [ 2808.406960] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2808.411291] ? do_raw_spin_unlock+0x75/0x190 [ 2808.417296] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2808.421110] ? do_wait_intr+0xf0/0xf0 [ 2808.423636] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2808.428006] mdc_close+0x53b/0xd70 [mdc] [ 2808.433988] lmv_close+0x18d/0x490 [lmv] [ 2808.438290] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2808.450439] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2808.456421] ll_md_close+0x274/0x800 [lustre] [ 2808.460801] ll_file_release+0x128/0xc50 [lustre] [ 2808.475061] __fput+0x10d/0x440 [ 2808.481092] ____fput+0x12/0x20 [ 2808.484576] task_work_run+0xc6/0x110 [ 2808.488866] exit_to_usermode_loop+0x1dd/0x1f0 [ 2808.492134] do_syscall_64+0x3d6/0x440 [ 2808.504974] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2808.516240] RIP: 0033:0x7f1a63983cd8 [ 2808.532501] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2808.554058] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2808.557050] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2808.559631] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2808.562494] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2808.565020] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2808.567670] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2819.555132] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 796s! [ 2819.565203] Showing busy workqueues and worker pools: [ 2819.571427] workqueue events: flags=0x0 [ 2819.576956] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2819.585077] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2819.592599] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2819.595559] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2819.608275] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2819.615914] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2819.622575] workqueue events_long: flags=0x0 [ 2819.625163] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2819.629194] pending: lu_objects_flush [obdclass] [ 2819.632248] workqueue events_unbound: flags=0x2 [ 2819.634804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2819.637551] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2819.643795] workqueue events_power_efficient: flags=0x80 [ 2819.648910] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2819.660698] pending: neigh_periodic_work, do_cache_clean [ 2819.665806] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2819.669810] pending: neigh_periodic_work, check_lifetime [ 2819.673548] workqueue mm_percpu_wq: flags=0x8 [ 2819.677277] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2819.682157] pending: vmstat_update [ 2819.684844] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2819.690138] pending: vmstat_update [ 2819.693274] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2819.698174] pending: vmstat_update [ 2819.701119] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2819.705010] pending: vmstat_update [ 2819.706889] workqueue writeback: flags=0x4a [ 2819.709171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2819.712744] pending: wb_workfn, wb_workfn [ 2819.715852] workqueue xprtiod: flags=0xa [ 2819.718117] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2819.721333] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2819.726256] workqueue ipv6_addrconf: flags=0x40008 [ 2819.727934] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2819.730748] pending: addrconf_verify_work [ 2819.732427] workqueue knbd0-recv: flags=0x1a [ 2819.734954] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2819.737801] in-flight: 647:recv_work [ 2819.739287] workqueue ptlrpc_pinger: flags=0x2 [ 2819.741223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2819.744220] pending: ptlrpc_pinger_main [ptlrpc] [ 2819.747819] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2809s workers=2 idle: 652 [ 2828.058041] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2828.060378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2828.074902] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2828.078318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2828.080902] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2828.082843] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2828.088442] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2828.090729] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2828.092953] RDX: 0000000000005c76 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2828.095114] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2828.097144] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2828.099311] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2828.101694] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2828.104126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2828.106065] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2828.108204] Call Trace: [ 2828.109139] [ 2828.109826] ? show_regs.cold.9+0x22/0x2f [ 2828.110965] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2828.112291] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2828.113906] ? watchdog+0x40/0x40 [ 2828.114919] ? hrtimer_interrupt+0x128/0x2c0 [ 2828.116315] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2828.117958] ? apic_timer_interrupt+0xf/0x20 [ 2828.119175] [ 2828.119965] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2828.121547] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2828.123217] do_raw_spin_lock+0x177/0x190 [ 2828.124645] _raw_spin_lock+0x21/0x30 [ 2828.125659] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2828.127436] obd_mod_free+0x46/0x140 [ptlrpc] [ 2828.129183] mdc_commit_open+0x91/0xb0 [mdc] [ 2828.130630] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2828.132523] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2828.134358] after_reply+0xe2c/0x1780 [ptlrpc] [ 2828.136286] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2828.138189] ? __next_timer_interrupt+0x160/0x160 [ 2828.139554] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2828.141156] ? do_wait_intr+0xf0/0xf0 [ 2828.142250] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2828.144356] kthread+0x1d1/0x200 [ 2828.145424] ? set_kthread_struct+0x70/0x70 [ 2828.146448] ret_from_fork+0x1f/0x30 [ 2832.056087] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2832.065639] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2832.110162] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2832.114842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2832.118061] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2832.120038] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2832.130100] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2832.132667] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2832.140562] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2832.149959] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2832.156690] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2832.162710] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2832.175857] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2832.189825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2832.202041] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2832.209187] Call Trace: [ 2832.211361] [ 2832.213715] ? show_regs.cold.9+0x22/0x2f [ 2832.222649] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2832.230246] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2832.233140] ? watchdog+0x40/0x40 [ 2832.236033] ? hrtimer_interrupt+0x128/0x2c0 [ 2832.238445] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2832.249589] ? apic_timer_interrupt+0xf/0x20 [ 2832.254340] [ 2832.255759] ? native_safe_halt+0x2e/0x50 [ 2832.260599] kvm_wait+0xcd/0xe0 [ 2832.263050] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2832.272682] do_raw_spin_lock+0x177/0x190 [ 2832.278732] _raw_spin_lock+0x21/0x30 [ 2832.286208] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2832.292908] ? do_raw_spin_unlock+0x75/0x190 [ 2832.298395] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2832.303645] ? do_wait_intr+0xf0/0xf0 [ 2832.308535] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2832.318640] mdc_close+0x53b/0xd70 [mdc] [ 2832.320847] lmv_close+0x18d/0x490 [lmv] [ 2832.331247] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2832.338257] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2832.341956] ll_md_close+0x274/0x800 [lustre] [ 2832.346308] ll_file_release+0x128/0xc50 [lustre] [ 2832.354784] __fput+0x10d/0x440 [ 2832.361832] ____fput+0x12/0x20 [ 2832.363734] task_work_run+0xc6/0x110 [ 2832.367170] exit_to_usermode_loop+0x1dd/0x1f0 [ 2832.371481] do_syscall_64+0x3d6/0x440 [ 2832.376892] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2832.379960] RIP: 0033:0x7f1a63983cd8 [ 2832.382169] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2832.396298] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2832.405042] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2832.412069] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2832.422179] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2832.430708] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2832.444798] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2836.057794] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 2836.061026] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2836.103317] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2836.109184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2836.113576] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2836.118797] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2836.128142] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2836.132562] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2836.139957] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2836.148400] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2836.157482] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2836.160128] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2836.168617] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2836.176149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2836.182837] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2836.191924] Call Trace: [ 2836.193184] [ 2836.194376] ? show_regs.cold.9+0x22/0x2f [ 2836.195781] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2836.199151] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2836.202163] ? watchdog+0x40/0x40 [ 2836.204776] ? hrtimer_interrupt+0x128/0x2c0 [ 2836.207176] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2836.209843] ? apic_timer_interrupt+0xf/0x20 [ 2836.211819] [ 2836.213049] ? native_safe_halt+0x2e/0x50 [ 2836.215315] kvm_wait+0xcd/0xe0 [ 2836.217782] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2836.221406] do_raw_spin_lock+0x177/0x190 [ 2836.227098] _raw_spin_lock+0x21/0x30 [ 2836.231735] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2836.238169] ? do_raw_spin_unlock+0x75/0x190 [ 2836.240349] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2836.247611] ? do_wait_intr+0xf0/0xf0 [ 2836.250628] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2836.255534] mdc_close+0x53b/0xd70 [mdc] [ 2836.257748] lmv_close+0x18d/0x490 [lmv] [ 2836.259897] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2836.263864] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2836.268608] ll_md_close+0x274/0x800 [lustre] [ 2836.271563] ll_file_release+0x128/0xc50 [lustre] [ 2836.274636] __fput+0x10d/0x440 [ 2836.278548] ____fput+0x12/0x20 [ 2836.282767] task_work_run+0xc6/0x110 [ 2836.284858] exit_to_usermode_loop+0x1dd/0x1f0 [ 2836.287652] do_syscall_64+0x3d6/0x440 [ 2836.290859] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2836.294550] RIP: 0033:0x7f1a63983cd8 [ 2836.296536] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2836.315315] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2836.321732] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2836.326176] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2836.330003] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2836.333412] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2836.336890] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2844.055154] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2844.059826] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2844.095805] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2844.103278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2844.105781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2844.107997] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2844.118190] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2844.122699] RAX: 0000000000007d0a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2844.126697] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2844.128794] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2844.133013] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2844.138503] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2844.144173] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2844.149764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2844.151565] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2844.153962] Call Trace: [ 2844.154621] [ 2844.155162] ? show_regs.cold.9+0x22/0x2f [ 2844.156962] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2844.159922] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2844.162823] ? watchdog+0x40/0x40 [ 2844.164143] ? hrtimer_interrupt+0x128/0x2c0 [ 2844.166794] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2844.169258] ? apic_timer_interrupt+0xf/0x20 [ 2844.172128] [ 2844.173155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2844.176750] do_raw_spin_lock+0x177/0x190 [ 2844.178672] _raw_spin_lock+0x21/0x30 [ 2844.179908] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2844.182176] ? do_raw_spin_unlock+0x75/0x190 [ 2844.184016] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2844.186159] ? do_wait_intr+0xf0/0xf0 [ 2844.187841] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2844.190067] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2844.192315] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2844.194871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2844.197226] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2844.199668] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2844.201592] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2844.204003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2844.205927] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2844.208208] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2844.210904] lmv_intent_open+0x338/0x1150 [lmv] [ 2844.214342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2844.216215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2844.218915] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2844.220887] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2844.223062] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2844.224924] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2844.226983] ? do_raw_spin_unlock+0x75/0x190 [ 2844.228826] ? _raw_spin_unlock+0x12/0x30 [ 2844.230044] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2844.232106] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2844.234207] ll_atomic_open+0x36a/0xd70 [lustre] [ 2844.236182] lookup_open+0x356/0xb40 [ 2844.237716] do_last+0x313/0xfc0 [ 2844.239028] ? nd_jump_root+0xe5/0x160 [ 2844.240578] ? path_init+0x437/0x520 [ 2844.241746] path_openat+0xf7/0x500 [ 2844.242895] ? do_raw_spin_unlock+0x75/0x190 [ 2844.244581] do_filp_open+0x99/0x140 [ 2844.245990] ? getname_flags+0x6e/0x330 [ 2844.247516] ? __check_object_size+0xff/0x256 [ 2844.248775] ? do_raw_spin_unlock+0x75/0x190 [ 2844.250121] ? _raw_spin_unlock+0x12/0x30 [ 2844.251728] do_sys_openat2+0x2b4/0x410 [ 2844.253191] do_sys_open+0x73/0xa0 [ 2844.254654] __x64_sys_openat+0x24/0x30 [ 2844.256193] do_syscall_64+0xc1/0x440 [ 2844.257680] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2844.259654] RIP: 0033:0x7f1a63983332 [ 2844.261040] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2844.268481] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2844.271511] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2844.274217] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2844.276855] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2844.279507] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2844.282328] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2849.761027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 826s! [ 2849.764858] Showing busy workqueues and worker pools: [ 2849.766773] workqueue events: flags=0x0 [ 2849.769916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2849.781273] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2849.790596] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2849.800855] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2849.808830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2849.815474] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2849.822198] workqueue events_long: flags=0x0 [ 2849.824602] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2849.827897] pending: lu_objects_flush [obdclass] [ 2849.830743] workqueue events_unbound: flags=0x2 [ 2849.833227] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2849.836445] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2849.842029] workqueue events_power_efficient: flags=0x80 [ 2849.844947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2849.848202] pending: neigh_periodic_work, do_cache_clean [ 2849.850817] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2849.853975] pending: neigh_periodic_work, check_lifetime [ 2849.860615] workqueue mm_percpu_wq: flags=0x8 [ 2849.863776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2849.867201] pending: vmstat_update [ 2849.869212] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2849.872563] pending: vmstat_update [ 2849.874565] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2849.878463] pending: vmstat_update [ 2849.880567] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2849.884587] pending: vmstat_update [ 2849.885759] workqueue writeback: flags=0x4a [ 2849.890075] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2849.894157] pending: wb_workfn, wb_workfn [ 2849.895730] workqueue xprtiod: flags=0xa [ 2849.897570] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2849.900740] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2849.905480] workqueue ipv6_addrconf: flags=0x40008 [ 2849.907531] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2849.910252] pending: addrconf_verify_work [ 2849.911912] workqueue knbd0-recv: flags=0x1a [ 2849.915025] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2849.919088] in-flight: 647:recv_work [ 2849.921831] workqueue ptlrpc_pinger: flags=0x2 [ 2849.923599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2849.926086] pending: ptlrpc_pinger_main [ptlrpc] [ 2849.932228] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2839s workers=2 idle: 652 [ 2856.062484] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2856.075321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2856.129255] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2856.147695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2856.157016] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2856.161850] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2856.180484] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2856.184725] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2856.190938] RDX: 00000000000037dc RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2856.200503] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2856.204204] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2856.213849] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2856.220803] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2856.232098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2856.236348] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2856.244098] Call Trace: [ 2856.248063] [ 2856.250391] ? show_regs.cold.9+0x22/0x2f [ 2856.256126] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2856.259787] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2856.266093] ? watchdog+0x40/0x40 [ 2856.269670] ? hrtimer_interrupt+0x128/0x2c0 [ 2856.276353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2856.278159] ? apic_timer_interrupt+0xf/0x20 [ 2856.286301] [ 2856.289217] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2856.292090] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2856.295561] do_raw_spin_lock+0x177/0x190 [ 2856.300496] _raw_spin_lock+0x21/0x30 [ 2856.303071] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2856.306112] obd_mod_free+0x46/0x140 [ptlrpc] [ 2856.308810] mdc_commit_open+0x91/0xb0 [mdc] [ 2856.311423] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2856.314500] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2856.318327] after_reply+0xe2c/0x1780 [ptlrpc] [ 2856.321323] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2856.325170] ? __next_timer_interrupt+0x160/0x160 [ 2856.327838] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2856.331809] ? do_wait_intr+0xf0/0xf0 [ 2856.334272] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2856.340239] kthread+0x1d1/0x200 [ 2856.342432] ? set_kthread_struct+0x70/0x70 [ 2856.344569] ret_from_fork+0x1f/0x30 [ 2860.057552] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2860.065947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2860.091422] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2860.096391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2860.100106] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2860.102803] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2860.111561] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2860.114793] RAX: 0000000000005786 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2860.118034] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2860.120991] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2860.124084] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2860.127271] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2860.130437] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2860.133888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2860.136439] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2860.139743] Call Trace: [ 2860.140761] [ 2860.141659] ? show_regs.cold.9+0x22/0x2f [ 2860.143600] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2860.146638] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2860.148955] ? watchdog+0x40/0x40 [ 2860.150443] ? hrtimer_interrupt+0x128/0x2c0 [ 2860.152580] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2860.154788] ? apic_timer_interrupt+0xf/0x20 [ 2860.156824] [ 2860.157762] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2860.160297] do_raw_spin_lock+0x177/0x190 [ 2860.162357] _raw_spin_lock+0x21/0x30 [ 2860.164271] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2860.166689] ? do_raw_spin_unlock+0x75/0x190 [ 2860.168992] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2860.171508] ? do_wait_intr+0xf0/0xf0 [ 2860.173024] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2860.175898] mdc_close+0x53b/0xd70 [mdc] [ 2860.178729] lmv_close+0x18d/0x490 [lmv] [ 2860.180498] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2860.182937] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2860.185017] ll_md_close+0x274/0x800 [lustre] [ 2860.186405] ll_file_release+0x128/0xc50 [lustre] [ 2860.187852] __fput+0x10d/0x440 [ 2860.189835] ____fput+0x12/0x20 [ 2860.191392] task_work_run+0xc6/0x110 [ 2860.193219] exit_to_usermode_loop+0x1dd/0x1f0 [ 2860.195021] do_syscall_64+0x3d6/0x440 [ 2860.196591] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2860.198730] RIP: 0033:0x7f1a63983cd8 [ 2860.200521] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2860.209556] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2860.212637] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2860.215806] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2860.218740] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2860.222128] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2860.225291] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2864.057012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2864.066958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2864.129736] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2864.136509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2864.146786] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2864.149735] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2864.163595] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2864.168608] RAX: 0000000000007286 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2864.171596] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2864.178554] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2864.181475] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2864.189592] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2864.192028] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2864.199516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2864.204360] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2864.216305] Call Trace: [ 2864.223471] [ 2864.224371] ? show_regs.cold.9+0x22/0x2f [ 2864.229090] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2864.231893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2864.238762] ? watchdog+0x40/0x40 [ 2864.241039] ? hrtimer_interrupt+0x128/0x2c0 [ 2864.249449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2864.251748] ? apic_timer_interrupt+0xf/0x20 [ 2864.254770] [ 2864.256402] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2864.268690] do_raw_spin_lock+0x177/0x190 [ 2864.275442] _raw_spin_lock+0x21/0x30 [ 2864.281844] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2864.288013] ? do_raw_spin_unlock+0x75/0x190 [ 2864.290032] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2864.292859] ? do_wait_intr+0xf0/0xf0 [ 2864.294935] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2864.297617] mdc_close+0x53b/0xd70 [mdc] [ 2864.299856] lmv_close+0x18d/0x490 [lmv] [ 2864.301793] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2864.312439] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2864.316388] ll_md_close+0x274/0x800 [lustre] [ 2864.319051] ll_file_release+0x128/0xc50 [lustre] [ 2864.327584] __fput+0x10d/0x440 [ 2864.333711] ____fput+0x12/0x20 [ 2864.341029] task_work_run+0xc6/0x110 [ 2864.351169] exit_to_usermode_loop+0x1dd/0x1f0 [ 2864.355800] do_syscall_64+0x3d6/0x440 [ 2864.362023] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2864.363811] RIP: 0033:0x7f1a63983cd8 [ 2864.380971] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2864.397555] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2864.405369] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2864.408412] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2864.413087] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2864.423689] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2864.432024] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2876.056846] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2876.065521] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2876.108683] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2876.117938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2876.127915] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.133957] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2876.143867] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2876.153811] RAX: 0000000000000886 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2876.163863] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2876.172347] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2876.174879] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2876.182990] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2876.193779] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2876.203866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2876.213200] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2876.217231] Call Trace: [ 2876.218693] [ 2876.220319] ? show_regs.cold.9+0x22/0x2f [ 2876.223740] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2876.226784] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2876.228697] ? watchdog+0x40/0x40 [ 2876.231333] ? hrtimer_interrupt+0x128/0x2c0 [ 2876.233413] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2876.237436] ? apic_timer_interrupt+0xf/0x20 [ 2876.239462] [ 2876.239993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.242733] do_raw_spin_lock+0x177/0x190 [ 2876.244973] _raw_spin_lock+0x21/0x30 [ 2876.253338] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2876.255734] ? do_raw_spin_unlock+0x75/0x190 [ 2876.261971] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2876.267986] ? do_wait_intr+0xf0/0xf0 [ 2876.272502] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2876.276455] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2876.284195] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2876.291838] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2876.301456] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2876.307987] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2876.314576] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2876.318528] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2876.326017] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2876.336654] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2876.341083] lmv_intent_open+0x338/0x1150 [lmv] [ 2876.343015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2876.347578] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2876.349818] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2876.355915] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2876.372952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2876.379613] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2876.385888] ? do_raw_spin_unlock+0x75/0x190 [ 2876.390695] ? _raw_spin_unlock+0x12/0x30 [ 2876.394617] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2876.398971] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2876.403443] ll_atomic_open+0x36a/0xd70 [lustre] [ 2876.409024] lookup_open+0x356/0xb40 [ 2876.410896] do_last+0x313/0xfc0 [ 2876.413783] ? nd_jump_root+0xe5/0x160 [ 2876.417419] ? path_init+0x437/0x520 [ 2876.420810] path_openat+0xf7/0x500 [ 2876.424593] ? do_raw_spin_unlock+0x75/0x190 [ 2876.430251] do_filp_open+0x99/0x140 [ 2876.432516] ? getname_flags+0x6e/0x330 [ 2876.437338] ? __check_object_size+0xff/0x256 [ 2876.442022] ? do_raw_spin_unlock+0x75/0x190 [ 2876.444661] ? _raw_spin_unlock+0x12/0x30 [ 2876.450599] do_sys_openat2+0x2b4/0x410 [ 2876.455472] do_sys_open+0x73/0xa0 [ 2876.463836] __x64_sys_openat+0x24/0x30 [ 2876.469719] do_syscall_64+0xc1/0x440 [ 2876.481219] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2876.485662] RIP: 0033:0x7f1a63983332 [ 2876.493257] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2876.518564] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2876.524973] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2876.531153] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2876.538285] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2876.550346] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2876.557600] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2879.979551] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 856s! [ 2879.984842] Showing busy workqueues and worker pools: [ 2879.987351] workqueue events: flags=0x0 [ 2879.994463] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2879.998894] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2880.004936] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2880.013281] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2880.024904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2880.027731] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2880.031856] workqueue events_long: flags=0x0 [ 2880.034201] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2880.037167] pending: lu_objects_flush [obdclass] [ 2880.039313] workqueue events_unbound: flags=0x2 [ 2880.041071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2880.043496] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2880.047953] workqueue events_power_efficient: flags=0x80 [ 2880.050371] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2880.052897] pending: neigh_periodic_work, do_cache_clean [ 2880.055084] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2880.057554] pending: neigh_periodic_work, check_lifetime [ 2880.059666] workqueue mm_percpu_wq: flags=0x8 [ 2880.060921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2880.063448] pending: vmstat_update [ 2880.064869] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2880.067409] pending: vmstat_update [ 2880.068870] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2880.071885] pending: vmstat_update [ 2880.073959] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2880.076944] pending: vmstat_update [ 2880.078592] workqueue writeback: flags=0x4a [ 2880.079690] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2880.081509] pending: wb_workfn, wb_workfn [ 2880.082581] workqueue xprtiod: flags=0xa [ 2880.083577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2880.085511] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2880.087526] workqueue ipv6_addrconf: flags=0x40008 [ 2880.089148] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2880.091587] pending: addrconf_verify_work [ 2880.093344] workqueue knbd0-recv: flags=0x1a [ 2880.095245] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2880.097728] in-flight: 647:recv_work [ 2880.099059] workqueue ptlrpc_pinger: flags=0x2 [ 2880.100332] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2880.102284] pending: ptlrpc_pinger_main [ptlrpc] [ 2880.104620] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2870s workers=2 idle: 652 [ 2884.061643] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 2884.071691] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2884.127936] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2884.138864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2884.149820] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2884.163328] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2884.183325] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2884.189926] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2884.194407] RDX: 0000000000007b2b RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2884.202164] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2884.217384] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2884.225672] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2884.234140] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2884.245614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2884.251888] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2884.263156] Call Trace: [ 2884.265130] [ 2884.266048] ? show_regs.cold.9+0x22/0x2f [ 2884.268577] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2884.270718] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2884.273582] ? watchdog+0x40/0x40 [ 2884.275154] ? hrtimer_interrupt+0x128/0x2c0 [ 2884.276854] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2884.279297] ? apic_timer_interrupt+0xf/0x20 [ 2884.281219] [ 2884.281986] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2884.284563] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2884.286706] do_raw_spin_lock+0x177/0x190 [ 2884.288130] _raw_spin_lock+0x21/0x30 [ 2884.289579] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2884.291715] obd_mod_free+0x46/0x140 [ptlrpc] [ 2884.293782] mdc_commit_open+0x91/0xb0 [mdc] [ 2884.295747] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2884.298248] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2884.301041] after_reply+0xe2c/0x1780 [ptlrpc] [ 2884.303208] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2884.304769] ? __next_timer_interrupt+0x160/0x160 [ 2884.306078] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2884.311323] ? do_wait_intr+0xf0/0xf0 [ 2884.312775] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2884.315307] kthread+0x1d1/0x200 [ 2884.316645] ? set_kthread_struct+0x70/0x70 [ 2884.318180] ret_from_fork+0x1f/0x30 [ 2888.060878] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2888.071618] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2888.121915] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2888.130896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2888.136320] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2888.138873] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2888.149349] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2888.152849] RAX: 0000000000000693 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2888.157280] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2888.162780] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2888.165779] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2888.176291] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2888.181172] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2888.195885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2888.201191] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2888.206228] Call Trace: [ 2888.208534] [ 2888.209301] ? show_regs.cold.9+0x22/0x2f [ 2888.210752] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2888.215785] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2888.217955] ? watchdog+0x40/0x40 [ 2888.219834] ? hrtimer_interrupt+0x128/0x2c0 [ 2888.224299] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2888.231576] ? apic_timer_interrupt+0xf/0x20 [ 2888.235240] [ 2888.236532] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2888.239543] do_raw_spin_lock+0x177/0x190 [ 2888.242872] _raw_spin_lock+0x21/0x30 [ 2888.248948] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2888.254763] ? do_raw_spin_unlock+0x75/0x190 [ 2888.259858] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2888.268189] ? do_wait_intr+0xf0/0xf0 [ 2888.270717] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2888.275841] mdc_close+0x53b/0xd70 [mdc] [ 2888.278188] lmv_close+0x18d/0x490 [lmv] [ 2888.288161] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2888.296704] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2888.299102] ll_md_close+0x274/0x800 [lustre] [ 2888.300669] ll_file_release+0x128/0xc50 [lustre] [ 2888.318657] __fput+0x10d/0x440 [ 2888.319719] ____fput+0x12/0x20 [ 2888.329248] task_work_run+0xc6/0x110 [ 2888.330596] exit_to_usermode_loop+0x1dd/0x1f0 [ 2888.335472] do_syscall_64+0x3d6/0x440 [ 2888.347944] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2888.351666] RIP: 0033:0x7f1a63983cd8 [ 2888.355556] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2888.381434] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2888.383773] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2888.387496] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2888.410480] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2888.412706] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2888.428540] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2892.057011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2892.064613] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2892.140211] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2892.160088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2892.175590] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2892.194190] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2892.222972] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2892.233554] RAX: 0000000000001e8a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2892.240820] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2892.256176] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2892.271567] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2892.283291] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2892.285791] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2892.307652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2892.321044] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2892.344101] Call Trace: [ 2892.351701] [ 2892.352449] ? show_regs.cold.9+0x22/0x2f [ 2892.353842] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2892.377348] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2892.384215] ? watchdog+0x40/0x40 [ 2892.385447] ? hrtimer_interrupt+0x128/0x2c0 [ 2892.393744] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2892.398323] ? apic_timer_interrupt+0xf/0x20 [ 2892.402943] [ 2892.405926] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2892.412500] do_raw_spin_lock+0x177/0x190 [ 2892.414996] _raw_spin_lock+0x21/0x30 [ 2892.417049] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2892.420826] ? do_raw_spin_unlock+0x75/0x190 [ 2892.424306] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2892.428480] ? do_wait_intr+0xf0/0xf0 [ 2892.432273] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2892.436764] mdc_close+0x53b/0xd70 [mdc] [ 2892.439026] lmv_close+0x18d/0x490 [lmv] [ 2892.441066] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2892.443975] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2892.446092] ll_md_close+0x274/0x800 [lustre] [ 2892.448580] ll_file_release+0x128/0xc50 [lustre] [ 2892.451732] __fput+0x10d/0x440 [ 2892.453086] ____fput+0x12/0x20 [ 2892.456049] task_work_run+0xc6/0x110 [ 2892.457792] exit_to_usermode_loop+0x1dd/0x1f0 [ 2892.460201] do_syscall_64+0x3d6/0x440 [ 2892.461560] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2892.463616] RIP: 0033:0x7f1a63983cd8 [ 2892.465949] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2892.474188] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2892.483035] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2892.484990] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2892.494493] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2892.498118] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2892.507278] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2908.061714] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 2908.066832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2908.102165] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2908.107793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2908.116849] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2908.119223] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2908.131514] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2908.149854] RAX: 0000000000007c06 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2908.154608] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2908.159071] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2908.162570] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2908.167532] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2908.178518] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2908.185634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2908.193451] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2908.200751] Call Trace: [ 2908.201541] [ 2908.202474] ? show_regs.cold.9+0x22/0x2f [ 2908.204904] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2908.207560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2908.211909] ? watchdog+0x40/0x40 [ 2908.213954] ? hrtimer_interrupt+0x128/0x2c0 [ 2908.215551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2908.217836] ? apic_timer_interrupt+0xf/0x20 [ 2908.220544] [ 2908.221273] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2908.225667] do_raw_spin_lock+0x177/0x190 [ 2908.227201] _raw_spin_lock+0x21/0x30 [ 2908.228365] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2908.230705] ? do_raw_spin_unlock+0x75/0x190 [ 2908.233163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2908.237220] ? do_wait_intr+0xf0/0xf0 [ 2908.238517] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2908.241391] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2908.243363] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2908.248237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2908.252609] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2908.255543] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2908.258650] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2908.262642] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2908.268563] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2908.270995] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2908.273707] lmv_intent_open+0x338/0x1150 [lmv] [ 2908.275996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2908.278650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2908.280883] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2908.282647] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2908.288662] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2908.293803] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2908.299074] ? do_raw_spin_unlock+0x75/0x190 [ 2908.303083] ? _raw_spin_unlock+0x12/0x30 [ 2908.304813] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2908.306631] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2908.309426] ll_atomic_open+0x36a/0xd70 [lustre] [ 2908.316557] lookup_open+0x356/0xb40 [ 2908.320366] do_last+0x313/0xfc0 [ 2908.322027] ? nd_jump_root+0xe5/0x160 [ 2908.331159] ? path_init+0x437/0x520 [ 2908.332410] path_openat+0xf7/0x500 [ 2908.333610] ? do_raw_spin_unlock+0x75/0x190 [ 2908.343567] do_filp_open+0x99/0x140 [ 2908.344798] ? getname_flags+0x6e/0x330 [ 2908.355398] ? __check_object_size+0xff/0x256 [ 2908.361958] ? do_raw_spin_unlock+0x75/0x190 [ 2908.364889] ? _raw_spin_unlock+0x12/0x30 [ 2908.371631] do_sys_openat2+0x2b4/0x410 [ 2908.374662] do_sys_open+0x73/0xa0 [ 2908.379846] __x64_sys_openat+0x24/0x30 [ 2908.386565] do_syscall_64+0xc1/0x440 [ 2908.391704] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2908.395459] RIP: 0033:0x7f1a63983332 [ 2908.409239] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2908.421310] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2908.424899] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2908.428879] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2908.432409] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2908.436146] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2908.439856] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2910.194474] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 887s! [ 2910.206981] Showing busy workqueues and worker pools: [ 2910.211655] workqueue events: flags=0x0 [ 2910.213098] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2910.219243] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2910.239190] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2910.249260] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2910.263008] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2910.277029] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2910.292596] workqueue events_long: flags=0x0 [ 2910.294095] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2910.297906] pending: lu_objects_flush [obdclass] [ 2910.302008] workqueue events_unbound: flags=0x2 [ 2910.304051] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2910.310619] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2910.324412] workqueue events_power_efficient: flags=0x80 [ 2910.327305] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2910.334011] pending: neigh_periodic_work, do_cache_clean [ 2910.338421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2910.342066] pending: neigh_periodic_work, check_lifetime [ 2910.347534] workqueue mm_percpu_wq: flags=0x8 [ 2910.349830] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2910.356956] pending: vmstat_update [ 2910.359151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2910.364072] pending: vmstat_update [ 2910.365538] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2910.374147] pending: vmstat_update [ 2910.380803] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2910.389380] pending: vmstat_update [ 2910.391300] workqueue writeback: flags=0x4a [ 2910.395604] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2910.399613] pending: wb_workfn, wb_workfn [ 2910.403169] workqueue xprtiod: flags=0xa [ 2910.405966] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2910.409733] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2910.416345] workqueue ipv6_addrconf: flags=0x40008 [ 2910.419014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2910.425376] pending: addrconf_verify_work [ 2910.428068] workqueue knbd0-recv: flags=0x1a [ 2910.430349] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2910.434822] in-flight: 647:recv_work [ 2910.437033] workqueue ptlrpc_pinger: flags=0x2 [ 2910.439020] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2910.444018] pending: ptlrpc_pinger_main [ptlrpc] [ 2910.448880] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2900s workers=2 idle: 652 [ 2912.058961] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 2912.070142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2912.133259] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2912.153132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2912.164472] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2912.168629] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2912.189540] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2912.197240] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2912.201892] RDX: 0000000000004051 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2912.210510] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2912.217482] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2912.229593] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2912.239496] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2912.245051] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2912.249074] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2912.255870] Call Trace: [ 2912.258978] [ 2912.260572] ? show_regs.cold.9+0x22/0x2f [ 2912.265541] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2912.272081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2912.275636] ? watchdog+0x40/0x40 [ 2912.278523] ? hrtimer_interrupt+0x128/0x2c0 [ 2912.285945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2912.292519] ? apic_timer_interrupt+0xf/0x20 [ 2912.294553] [ 2912.295358] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2912.302043] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2912.308049] do_raw_spin_lock+0x177/0x190 [ 2912.311933] _raw_spin_lock+0x21/0x30 [ 2912.314236] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2912.319386] obd_mod_free+0x46/0x140 [ptlrpc] [ 2912.320876] mdc_commit_open+0x91/0xb0 [mdc] [ 2912.324966] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2912.333398] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2912.337205] after_reply+0xe2c/0x1780 [ptlrpc] [ 2912.340053] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2912.345589] ? __next_timer_interrupt+0x160/0x160 [ 2912.349076] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2912.352997] ? do_wait_intr+0xf0/0xf0 [ 2912.356865] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2912.360496] kthread+0x1d1/0x200 [ 2912.363282] ? set_kthread_struct+0x70/0x70 [ 2912.367953] ret_from_fork+0x1f/0x30 [ 2916.056334] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2916.066209] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2916.113556] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2916.118449] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2916.125694] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2916.133062] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2916.151167] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2916.161396] RAX: 00000000000037d0 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2916.170290] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2916.177587] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2916.182208] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2916.186430] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2916.190847] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2916.205469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2916.210934] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2916.219973] Call Trace: [ 2916.221071] [ 2916.225136] ? show_regs.cold.9+0x22/0x2f [ 2916.231092] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2916.234635] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2916.236838] ? watchdog+0x40/0x40 [ 2916.237861] ? hrtimer_interrupt+0x128/0x2c0 [ 2916.240463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2916.242978] ? apic_timer_interrupt+0xf/0x20 [ 2916.247956] [ 2916.248658] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2916.252199] do_raw_spin_lock+0x177/0x190 [ 2916.259462] _raw_spin_lock+0x21/0x30 [ 2916.266514] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2916.272319] ? do_raw_spin_unlock+0x75/0x190 [ 2916.280215] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2916.283612] ? do_wait_intr+0xf0/0xf0 [ 2916.285282] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2916.297067] mdc_close+0x53b/0xd70 [mdc] [ 2916.299360] lmv_close+0x18d/0x490 [lmv] [ 2916.301445] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2916.303480] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2916.312181] ll_md_close+0x274/0x800 [lustre] [ 2916.313952] ll_file_release+0x128/0xc50 [lustre] [ 2916.320167] __fput+0x10d/0x440 [ 2916.321512] ____fput+0x12/0x20 [ 2916.323413] task_work_run+0xc6/0x110 [ 2916.329492] exit_to_usermode_loop+0x1dd/0x1f0 [ 2916.338182] do_syscall_64+0x3d6/0x440 [ 2916.339511] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2916.344402] RIP: 0033:0x7f1a63983cd8 [ 2916.354239] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2916.369324] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2916.379401] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2916.387221] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2916.395491] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2916.404700] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2916.417316] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2920.057024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2920.061982] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2920.120184] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2920.135399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2920.151634] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2920.154953] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2920.165294] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2920.172929] RAX: 0000000000003e82 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2920.184138] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2920.189894] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2920.200084] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2920.215357] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2920.224549] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2920.228513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2920.237347] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2920.246407] Call Trace: [ 2920.247376] [ 2920.251811] ? show_regs.cold.9+0x22/0x2f [ 2920.253312] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2920.259420] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2920.268210] ? watchdog+0x40/0x40 [ 2920.273160] ? hrtimer_interrupt+0x128/0x2c0 [ 2920.284261] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2920.287417] ? apic_timer_interrupt+0xf/0x20 [ 2920.292055] [ 2920.298278] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2920.303927] do_raw_spin_lock+0x177/0x190 [ 2920.310024] _raw_spin_lock+0x21/0x30 [ 2920.314019] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2920.316876] ? do_raw_spin_unlock+0x75/0x190 [ 2920.320090] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2920.324426] ? do_wait_intr+0xf0/0xf0 [ 2920.325837] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2920.332704] mdc_close+0x53b/0xd70 [mdc] [ 2920.334179] lmv_close+0x18d/0x490 [lmv] [ 2920.338384] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2920.345515] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2920.350756] ll_md_close+0x274/0x800 [lustre] [ 2920.356115] ll_file_release+0x128/0xc50 [lustre] [ 2920.359875] __fput+0x10d/0x440 [ 2920.363014] ____fput+0x12/0x20 [ 2920.366119] task_work_run+0xc6/0x110 [ 2920.368699] exit_to_usermode_loop+0x1dd/0x1f0 [ 2920.373700] do_syscall_64+0x3d6/0x440 [ 2920.376765] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2920.384831] RIP: 0033:0x7f1a63983cd8 [ 2920.387842] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2920.402207] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2920.413133] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2920.417102] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2920.427044] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2920.432846] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2920.436642] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2936.060832] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 2936.067367] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2936.126328] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2936.149886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2936.169324] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2936.179575] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2936.197958] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2936.205940] RAX: 0000000000006668 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2936.213222] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2936.238586] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2936.259873] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2936.273618] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2936.292094] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2936.303415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2936.312267] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2936.319223] Call Trace: [ 2936.323113] [ 2936.324327] ? show_regs.cold.9+0x22/0x2f [ 2936.329868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2936.333522] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2936.342697] ? watchdog+0x40/0x40 [ 2936.343862] ? hrtimer_interrupt+0x128/0x2c0 [ 2936.349164] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2936.355619] ? apic_timer_interrupt+0xf/0x20 [ 2936.359936] [ 2936.361029] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2936.366830] do_raw_spin_lock+0x177/0x190 [ 2936.372441] _raw_spin_lock+0x21/0x30 [ 2936.374791] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2936.378410] ? do_raw_spin_unlock+0x75/0x190 [ 2936.381373] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2936.392371] ? do_wait_intr+0xf0/0xf0 [ 2936.395027] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2936.403959] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2936.407624] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2936.414249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2936.421953] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2936.431614] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2936.437841] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2936.444936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2936.451326] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2936.456606] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2936.460061] lmv_intent_open+0x338/0x1150 [lmv] [ 2936.465988] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2936.469240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2936.474831] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2936.477424] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2936.482751] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2936.487054] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2936.490075] ? do_raw_spin_unlock+0x75/0x190 [ 2936.493208] ? _raw_spin_unlock+0x12/0x30 [ 2936.495137] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2936.502435] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2936.507428] ll_atomic_open+0x36a/0xd70 [lustre] [ 2936.514260] lookup_open+0x356/0xb40 [ 2936.524833] do_last+0x313/0xfc0 [ 2936.527090] ? nd_jump_root+0xe5/0x160 [ 2936.529828] ? path_init+0x437/0x520 [ 2936.533745] path_openat+0xf7/0x500 [ 2936.535858] ? do_raw_spin_unlock+0x75/0x190 [ 2936.538508] do_filp_open+0x99/0x140 [ 2936.540873] ? getname_flags+0x6e/0x330 [ 2936.544893] ? __check_object_size+0xff/0x256 [ 2936.551620] ? do_raw_spin_unlock+0x75/0x190 [ 2936.556866] ? _raw_spin_unlock+0x12/0x30 [ 2936.561160] do_sys_openat2+0x2b4/0x410 [ 2936.564941] do_sys_open+0x73/0xa0 [ 2936.566915] __x64_sys_openat+0x24/0x30 [ 2936.569545] do_syscall_64+0xc1/0x440 [ 2936.571872] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2936.577199] RIP: 0033:0x7f1a63983332 [ 2936.579841] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2936.596180] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2936.608101] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2936.618021] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2936.622286] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2936.628436] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2936.632753] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2940.059260] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 2940.066798] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2940.093790] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2940.099629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2940.103665] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2940.107052] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2940.117207] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2940.121534] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2940.125943] RDX: 000000000000388b RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2940.131824] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2940.138525] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2940.142061] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2940.144871] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2940.148124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2940.151258] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2940.155651] Call Trace: [ 2940.157214] [ 2940.158134] ? show_regs.cold.9+0x22/0x2f [ 2940.160497] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2940.162686] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2940.164390] ? watchdog+0x40/0x40 [ 2940.165818] ? hrtimer_interrupt+0x128/0x2c0 [ 2940.167658] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2940.170127] ? apic_timer_interrupt+0xf/0x20 [ 2940.172336] [ 2940.173290] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2940.176314] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2940.178938] do_raw_spin_lock+0x177/0x190 [ 2940.182381] _raw_spin_lock+0x21/0x30 [ 2940.184739] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2940.188547] obd_mod_free+0x46/0x140 [ptlrpc] [ 2940.192091] mdc_commit_open+0x91/0xb0 [mdc] [ 2940.194581] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2940.198332] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2940.202075] after_reply+0xe2c/0x1780 [ptlrpc] [ 2940.205403] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2940.208659] ? __next_timer_interrupt+0x160/0x160 [ 2940.211219] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2940.214046] ? do_wait_intr+0xf0/0xf0 [ 2940.216403] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2940.220115] kthread+0x1d1/0x200 [ 2940.221593] ? set_kthread_struct+0x70/0x70 [ 2940.224553] ret_from_fork+0x1f/0x30 [ 2940.897137] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 917s! [ 2940.900084] Showing busy workqueues and worker pools: [ 2940.909026] workqueue events: flags=0x0 [ 2940.916656] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2940.922092] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2940.927221] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2940.930719] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2940.939855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2940.941679] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2940.948339] workqueue events_long: flags=0x0 [ 2940.950962] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2940.958287] pending: lu_objects_flush [obdclass] [ 2940.962247] workqueue events_unbound: flags=0x2 [ 2940.965222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2940.970591] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2940.978192] workqueue events_power_efficient: flags=0x80 [ 2940.979685] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2940.984362] pending: neigh_periodic_work, do_cache_clean [ 2940.985910] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2940.997092] pending: neigh_periodic_work, check_lifetime [ 2941.005599] workqueue mm_percpu_wq: flags=0x8 [ 2941.007337] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2941.013320] pending: vmstat_update [ 2941.019078] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2941.023260] pending: vmstat_update [ 2941.029224] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2941.036188] pending: vmstat_update [ 2941.037766] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2941.040734] pending: vmstat_update [ 2941.046174] workqueue writeback: flags=0x4a [ 2941.048700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2941.056129] pending: wb_workfn, wb_workfn [ 2941.061019] workqueue xprtiod: flags=0xa [ 2941.065181] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2941.071432] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2941.078358] workqueue ipv6_addrconf: flags=0x40008 [ 2941.082769] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2941.088782] pending: addrconf_verify_work [ 2941.094243] workqueue knbd0-recv: flags=0x1a [ 2941.099304] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2941.103590] in-flight: 647:recv_work [ 2941.105354] workqueue ptlrpc_pinger: flags=0x2 [ 2941.110077] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2941.118554] pending: ptlrpc_pinger_main [ptlrpc] [ 2941.120777] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2931s workers=2 idle: 652 [ 2944.058354] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2944.075076] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2944.137456] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2944.152729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2944.169432] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2944.176782] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2944.206083] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2944.218081] RAX: 0000000000000486 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2944.227433] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2944.234701] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2944.244257] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2944.252322] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2944.260206] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2944.268538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2944.276763] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2944.290866] Call Trace: [ 2944.294180] [ 2944.297839] ? show_regs.cold.9+0x22/0x2f [ 2944.301909] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2944.310549] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2944.313407] ? watchdog+0x40/0x40 [ 2944.316309] ? hrtimer_interrupt+0x128/0x2c0 [ 2944.321612] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2944.325859] ? apic_timer_interrupt+0xf/0x20 [ 2944.329218] [ 2944.330888] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2944.333748] do_raw_spin_lock+0x177/0x190 [ 2944.343384] _raw_spin_lock+0x21/0x30 [ 2944.347064] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2944.355328] ? do_raw_spin_unlock+0x75/0x190 [ 2944.362545] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2944.371322] ? do_wait_intr+0xf0/0xf0 [ 2944.375611] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2944.383810] mdc_close+0x53b/0xd70 [mdc] [ 2944.390357] lmv_close+0x18d/0x490 [lmv] [ 2944.397146] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2944.401541] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2944.411200] ll_md_close+0x274/0x800 [lustre] [ 2944.419172] ll_file_release+0x128/0xc50 [lustre] [ 2944.423790] __fput+0x10d/0x440 [ 2944.430802] ____fput+0x12/0x20 [ 2944.432743] task_work_run+0xc6/0x110 [ 2944.437235] exit_to_usermode_loop+0x1dd/0x1f0 [ 2944.439979] do_syscall_64+0x3d6/0x440 [ 2944.447159] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2944.453150] RIP: 0033:0x7f1a63983cd8 [ 2944.455235] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2944.468467] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2944.480920] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2944.485642] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2944.491937] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2944.497325] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2944.504329] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2948.059962] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2948.075128] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2948.117849] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2948.122156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2948.126471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2948.131814] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2948.152733] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2948.164778] RAX: 0000000000002034 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2948.169999] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2948.178881] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2948.182951] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2948.188003] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2948.193090] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2948.200415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2948.207959] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2948.210615] Call Trace: [ 2948.222934] [ 2948.224118] ? show_regs.cold.9+0x22/0x2f [ 2948.232243] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2948.239945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2948.250154] ? watchdog+0x40/0x40 [ 2948.254998] ? hrtimer_interrupt+0x128/0x2c0 [ 2948.260464] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2948.263028] ? apic_timer_interrupt+0xf/0x20 [ 2948.265735] [ 2948.266532] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2948.273937] do_raw_spin_lock+0x177/0x190 [ 2948.276146] _raw_spin_lock+0x21/0x30 [ 2948.279990] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2948.282208] ? do_raw_spin_unlock+0x75/0x190 [ 2948.293204] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2948.298026] ? do_wait_intr+0xf0/0xf0 [ 2948.301026] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2948.304512] mdc_close+0x53b/0xd70 [mdc] [ 2948.306506] lmv_close+0x18d/0x490 [lmv] [ 2948.308543] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2948.311536] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2948.313990] ll_md_close+0x274/0x800 [lustre] [ 2948.316311] ll_file_release+0x128/0xc50 [lustre] [ 2948.318833] __fput+0x10d/0x440 [ 2948.320578] ____fput+0x12/0x20 [ 2948.322153] task_work_run+0xc6/0x110 [ 2948.324187] exit_to_usermode_loop+0x1dd/0x1f0 [ 2948.326762] do_syscall_64+0x3d6/0x440 [ 2948.329349] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2948.338678] RIP: 0033:0x7f1a63983cd8 [ 2948.340077] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2948.357099] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2948.363546] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2948.367918] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2948.372159] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2948.376817] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2948.380755] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2968.053026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 2968.055086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2968.058095] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 2968.100990] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2968.101014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2968.101018] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2968.104506] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2968.104593] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2968.104601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2968.114912] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2968.114928] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2968.115088] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2968.115091] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2968.115094] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2968.115095] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2968.115097] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2968.115100] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2968.115103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2968.115105] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2968.115113] Call Trace: [ 2968.115120] [ 2968.115127] ? show_regs.cold.9+0x22/0x2f [ 2968.115158] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2968.115164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2968.115168] ? watchdog+0x40/0x40 [ 2968.115171] ? hrtimer_interrupt+0x128/0x2c0 [ 2968.115176] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2968.115184] ? apic_timer_interrupt+0xf/0x20 [ 2968.115189] [ 2968.115192] ? native_safe_halt+0x2e/0x50 [ 2968.115196] kvm_wait+0xcd/0xe0 [ 2968.115205] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2968.115216] do_raw_spin_lock+0x177/0x190 [ 2968.115221] _raw_spin_lock+0x21/0x30 [ 2968.115225] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2968.115509] ? do_raw_spin_unlock+0x75/0x190 [ 2968.115516] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2968.115718] ? do_wait_intr+0xf0/0xf0 [ 2968.115728] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2968.116013] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2968.116256] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2968.116474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2968.116855] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2968.116899] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2968.116927] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2968.116963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2968.117050] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2968.117380] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2968.117410] lmv_intent_open+0x338/0x1150 [lmv] [ 2968.117448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2968.117546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2968.117638] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2968.117676] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2968.117774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2968.117880] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2968.118099] ? do_raw_spin_unlock+0x75/0x190 [ 2968.118112] ? _raw_spin_unlock+0x12/0x30 [ 2968.122934] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2968.122968] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2968.122973] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2968.122979] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2968.122982] RDX: 00000000000028c6 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 2968.122984] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 2968.122986] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 2968.122989] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2968.122991] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 2968.122994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2968.122997] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 2968.123010] Call Trace: [ 2968.123015] [ 2968.123020] ? show_regs.cold.9+0x22/0x2f [ 2968.123029] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2968.123035] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2968.123039] ? watchdog+0x40/0x40 [ 2968.123044] ? hrtimer_interrupt+0x128/0x2c0 [ 2968.123049] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2968.123058] ? apic_timer_interrupt+0xf/0x20 [ 2968.123063] [ 2968.123066] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2968.123071] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2968.123075] do_raw_spin_lock+0x177/0x190 [ 2968.123079] _raw_spin_lock+0x21/0x30 [ 2968.123086] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2968.123406] obd_mod_free+0x46/0x140 [ptlrpc] [ 2968.123626] mdc_commit_open+0x91/0xb0 [mdc] [ 2968.123672] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2968.124026] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2968.124249] after_reply+0xe2c/0x1780 [ptlrpc] [ 2968.124464] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2968.124696] ? __next_timer_interrupt+0x160/0x160 [ 2968.124710] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2968.125040] ? do_wait_intr+0xf0/0xf0 [ 2968.125049] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2968.125349] kthread+0x1d1/0x200 [ 2968.125354] ? set_kthread_struct+0x70/0x70 [ 2968.125363] ret_from_fork+0x1f/0x30 [ 2968.419383] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2968.424339] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2968.428771] ll_atomic_open+0x36a/0xd70 [lustre] [ 2968.434771] lookup_open+0x356/0xb40 [ 2968.439996] do_last+0x313/0xfc0 [ 2968.443860] ? nd_jump_root+0xe5/0x160 [ 2968.447922] ? path_init+0x437/0x520 [ 2968.452356] path_openat+0xf7/0x500 [ 2968.455321] ? do_raw_spin_unlock+0x75/0x190 [ 2968.459362] do_filp_open+0x99/0x140 [ 2968.461924] ? getname_flags+0x6e/0x330 [ 2968.464081] ? __check_object_size+0xff/0x256 [ 2968.466321] ? do_raw_spin_unlock+0x75/0x190 [ 2968.471118] ? _raw_spin_unlock+0x12/0x30 [ 2968.473610] do_sys_openat2+0x2b4/0x410 [ 2968.475738] do_sys_open+0x73/0xa0 [ 2968.477666] __x64_sys_openat+0x24/0x30 [ 2968.482270] do_syscall_64+0xc1/0x440 [ 2968.484275] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2968.487386] RIP: 0033:0x7f1a63983332 [ 2968.489718] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2968.499387] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2968.503966] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2968.507497] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2968.511099] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2968.516369] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2968.524743] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2971.616211] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 948s! [ 2971.619481] Showing busy workqueues and worker pools: [ 2971.624512] workqueue events: flags=0x0 [ 2971.625836] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2971.631694] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 2971.636632] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2971.641507] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 2971.650573] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2971.655637] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 2971.662239] workqueue events_long: flags=0x0 [ 2971.667523] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2971.670547] pending: lu_objects_flush [obdclass] [ 2971.673256] workqueue events_unbound: flags=0x2 [ 2971.675622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2971.681560] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2971.688556] workqueue events_power_efficient: flags=0x80 [ 2971.692592] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2971.697245] pending: neigh_periodic_work, do_cache_clean [ 2971.701463] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2971.705304] pending: neigh_periodic_work, check_lifetime [ 2971.709274] workqueue mm_percpu_wq: flags=0x8 [ 2971.712570] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2971.717382] pending: vmstat_update [ 2971.720051] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2971.723717] pending: vmstat_update [ 2971.725876] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2971.729592] pending: vmstat_update [ 2971.731939] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2971.736170] pending: vmstat_update [ 2971.738294] workqueue writeback: flags=0x4a [ 2971.740944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2971.745065] pending: wb_workfn, wb_workfn [ 2971.747892] workqueue xprtiod: flags=0xa [ 2971.749764] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2971.753565] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 2971.758551] workqueue ipv6_addrconf: flags=0x40008 [ 2971.762236] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2971.766538] pending: addrconf_verify_work [ 2971.769575] workqueue knbd0-recv: flags=0x1a [ 2971.772095] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2971.775363] in-flight: 647:recv_work [ 2971.778067] workqueue ptlrpc_pinger: flags=0x2 [ 2971.780540] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2971.784137] pending: ptlrpc_pinger_main [ptlrpc] [ 2971.787456] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2961s workers=2 idle: 652 [ 2972.056023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 2972.058770] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2972.080249] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2972.085066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2972.089453] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2972.091288] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2972.100191] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2972.102918] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2972.105886] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 2972.109127] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2972.112261] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 2972.115040] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 2972.117791] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 2972.121024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2972.122849] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 2972.125513] Call Trace: [ 2972.130258] [ 2972.130992] ? show_regs.cold.9+0x22/0x2f [ 2972.132960] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2972.136128] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2972.143800] ? watchdog+0x40/0x40 [ 2972.148026] ? hrtimer_interrupt+0x128/0x2c0 [ 2972.151333] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2972.152848] ? apic_timer_interrupt+0xf/0x20 [ 2972.157460] [ 2972.158452] ? native_safe_halt+0x2e/0x50 [ 2972.159547] kvm_wait+0xcd/0xe0 [ 2972.162381] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2972.164365] do_raw_spin_lock+0x177/0x190 [ 2972.165665] _raw_spin_lock+0x21/0x30 [ 2972.169208] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2972.172011] ? do_raw_spin_unlock+0x75/0x190 [ 2972.176585] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2972.181309] ? do_wait_intr+0xf0/0xf0 [ 2972.182901] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2972.186173] mdc_close+0x53b/0xd70 [mdc] [ 2972.194894] lmv_close+0x18d/0x490 [lmv] [ 2972.198910] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2972.202378] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2972.207933] ll_md_close+0x274/0x800 [lustre] [ 2972.217033] ll_file_release+0x128/0xc50 [lustre] [ 2972.223527] __fput+0x10d/0x440 [ 2972.225299] ____fput+0x12/0x20 [ 2972.227423] task_work_run+0xc6/0x110 [ 2972.229691] exit_to_usermode_loop+0x1dd/0x1f0 [ 2972.232652] do_syscall_64+0x3d6/0x440 [ 2972.233963] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2972.236531] RIP: 0033:0x7f1a63983cd8 [ 2972.238056] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2972.245821] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2972.249046] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 2972.252053] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2972.254799] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 2972.259410] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 2972.261998] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2976.059257] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 2976.063528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2976.082019] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2976.087242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2976.091258] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2976.093260] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2976.100445] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2976.103498] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2976.106329] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 2976.109059] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 2976.111908] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 2976.114907] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 2976.118009] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 2976.121478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2976.124159] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 2976.127289] Call Trace: [ 2976.128373] [ 2976.129253] ? show_regs.cold.9+0x22/0x2f [ 2976.131278] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2976.133895] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2976.136026] ? watchdog+0x40/0x40 [ 2976.137494] ? hrtimer_interrupt+0x128/0x2c0 [ 2976.139367] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2976.142034] ? apic_timer_interrupt+0xf/0x20 [ 2976.143848] [ 2976.144881] ? native_safe_halt+0x2e/0x50 [ 2976.146922] kvm_wait+0xcd/0xe0 [ 2976.148462] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2976.150838] do_raw_spin_lock+0x177/0x190 [ 2976.152335] _raw_spin_lock+0x21/0x30 [ 2976.153569] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2976.156382] ? do_raw_spin_unlock+0x75/0x190 [ 2976.158586] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2976.161300] ? do_wait_intr+0xf0/0xf0 [ 2976.162904] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2976.165217] mdc_close+0x53b/0xd70 [mdc] [ 2976.167037] lmv_close+0x18d/0x490 [lmv] [ 2976.168919] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2976.171645] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2976.173910] ll_md_close+0x274/0x800 [lustre] [ 2976.176315] ll_file_release+0x128/0xc50 [lustre] [ 2976.178476] __fput+0x10d/0x440 [ 2976.179989] ____fput+0x12/0x20 [ 2976.181426] task_work_run+0xc6/0x110 [ 2976.182817] exit_to_usermode_loop+0x1dd/0x1f0 [ 2976.184661] do_syscall_64+0x3d6/0x440 [ 2976.186530] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2976.188846] RIP: 0033:0x7f1a63983cd8 [ 2976.190307] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2976.198659] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2976.201977] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 2976.204928] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 2976.208089] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 2976.211715] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 2976.214564] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2982.434017] rcu: INFO: rcu_sched self-detected stall on CPU [ 2982.439282] rcu: 3-...!: (554916 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 2982.446358] (t=960039 jiffies g=269633 q=545) [ 2982.449057] rcu: rcu_sched kthread starved for 959688 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 2982.455363] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2982.460476] rcu: RCU grace-period kthread stack dump: [ 2982.464947] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2982.471839] Call Trace: [ 2982.473177] __schedule+0x351/0xcb0 [ 2982.476991] schedule+0xc0/0x180 [ 2982.479776] schedule_timeout+0xb4/0x190 [ 2982.482143] ? do_raw_spin_unlock+0x75/0x190 [ 2982.487476] ? __next_timer_interrupt+0x160/0x160 [ 2982.490781] ? prepare_to_swait_event+0xf3/0x140 [ 2982.495299] rcu_gp_kthread+0x8dd/0x15b0 [ 2982.497753] ? force_qs_rnp+0x2c0/0x2c0 [ 2982.500850] kthread+0x1d1/0x200 [ 2982.501870] ? set_kthread_struct+0x70/0x70 [ 2982.503819] ret_from_fork+0x1f/0x30 [ 2982.509955] rcu: Stack dump where RCU GP kthread last ran: [ 2982.511790] NMI backtrace for cpu 3 [ 2982.513182] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2982.518753] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2982.525410] Call Trace: [ 2982.526841] [ 2982.530846] ? dump_stack+0xbb/0x10e [ 2982.533940] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2982.538867] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2982.541199] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2982.543955] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2982.548154] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2982.551632] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 2982.556102] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 2982.559152] ? update_process_times+0x6d/0xc0 [ 2982.561782] ? tick_sched_handle.isra.18+0x27/0x90 [ 2982.565916] ? tick_sched_timer+0x53/0xb0 [ 2982.568134] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2982.570384] ? tick_sched_handle.isra.18+0x90/0x90 [ 2982.573154] ? hrtimer_interrupt+0x128/0x2c0 [ 2982.575490] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2982.578218] ? apic_timer_interrupt+0xf/0x20 [ 2982.580377] [ 2982.581646] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2982.584744] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2982.588129] ? do_raw_spin_lock+0x177/0x190 [ 2982.590698] ? _raw_spin_lock+0x21/0x30 [ 2982.592858] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2982.596317] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2982.599455] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2982.601976] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2982.604653] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2982.607507] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2982.610814] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2982.614666] ? __next_timer_interrupt+0x160/0x160 [ 2982.617721] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2982.620406] ? do_wait_intr+0xf0/0xf0 [ 2982.622697] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2982.625898] ? kthread+0x1d1/0x200 [ 2982.629955] ? set_kthread_struct+0x70/0x70 [ 2982.633893] ? ret_from_fork+0x1f/0x30 [ 2982.638177] NMI backtrace for cpu 3 [ 2982.640838] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2982.647476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2982.652324] Call Trace: [ 2982.653775] [ 2982.654860] ? dump_stack+0xbb/0x10e [ 2982.656712] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2982.659436] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2982.661849] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2982.664652] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2982.667350] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2982.670255] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2982.672610] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2982.675172] ? update_process_times+0x6d/0xc0 [ 2982.677253] ? tick_sched_handle.isra.18+0x27/0x90 [ 2982.679917] ? tick_sched_timer+0x53/0xb0 [ 2982.682493] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2982.685134] ? tick_sched_handle.isra.18+0x90/0x90 [ 2982.688072] ? hrtimer_interrupt+0x128/0x2c0 [ 2982.690657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2982.693316] ? apic_timer_interrupt+0xf/0x20 [ 2982.696178] [ 2982.697856] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2982.701113] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2982.706810] ? do_raw_spin_lock+0x177/0x190 [ 2982.709544] ? _raw_spin_lock+0x21/0x30 [ 2982.711634] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2982.716738] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2982.718955] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2982.720867] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2982.723975] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2982.728318] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2982.731958] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2982.738195] ? __next_timer_interrupt+0x160/0x160 [ 2982.741953] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2982.745899] ? do_wait_intr+0xf0/0xf0 [ 2982.748867] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2982.751663] ? kthread+0x1d1/0x200 [ 2982.753222] ? set_kthread_struct+0x70/0x70 [ 2982.756167] ? ret_from_fork+0x1f/0x30 [ 2996.057659] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 2996.067235] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2996.150521] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2996.170528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2996.179311] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2996.189051] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2996.222552] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2996.233556] RAX: 0000000000007e8b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 2996.247616] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 2996.256503] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 2996.262302] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 2996.267397] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 2996.277309] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 2996.289027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2996.295464] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 2996.303941] Call Trace: [ 2996.305163] [ 2996.306479] ? show_regs.cold.9+0x22/0x2f [ 2996.309494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2996.314843] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2996.328070] ? watchdog+0x40/0x40 [ 2996.332162] ? hrtimer_interrupt+0x128/0x2c0 [ 2996.336562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2996.342177] ? apic_timer_interrupt+0xf/0x20 [ 2996.353768] [ 2996.361482] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2996.375485] do_raw_spin_lock+0x177/0x190 [ 2996.376977] _raw_spin_lock+0x21/0x30 [ 2996.390566] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2996.410641] ? do_raw_spin_unlock+0x75/0x190 [ 2996.412131] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2996.421189] ? do_wait_intr+0xf0/0xf0 [ 2996.429269] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2996.436660] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2996.445793] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2996.450857] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2996.453973] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2996.460512] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2996.463184] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2996.466019] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2996.468778] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2996.472491] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2996.475563] lmv_intent_open+0x338/0x1150 [lmv] [ 2996.478845] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2996.483444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2996.488522] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2996.490680] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2996.493378] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2996.496478] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2996.508587] ? do_raw_spin_unlock+0x75/0x190 [ 2996.514893] ? _raw_spin_unlock+0x12/0x30 [ 2996.525043] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2996.537397] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2996.545644] ll_atomic_open+0x36a/0xd70 [lustre] [ 2996.553382] lookup_open+0x356/0xb40 [ 2996.554724] do_last+0x313/0xfc0 [ 2996.564912] ? nd_jump_root+0xe5/0x160 [ 2996.566191] ? path_init+0x437/0x520 [ 2996.578106] path_openat+0xf7/0x500 [ 2996.581673] ? do_raw_spin_unlock+0x75/0x190 [ 2996.585379] do_filp_open+0x99/0x140 [ 2996.588074] ? getname_flags+0x6e/0x330 [ 2996.596682] ? __check_object_size+0xff/0x256 [ 2996.602157] ? do_raw_spin_unlock+0x75/0x190 [ 2996.608472] ? _raw_spin_unlock+0x12/0x30 [ 2996.619385] do_sys_openat2+0x2b4/0x410 [ 2996.621747] do_sys_open+0x73/0xa0 [ 2996.624931] __x64_sys_openat+0x24/0x30 [ 2996.630647] do_syscall_64+0xc1/0x440 [ 2996.641548] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2996.651624] RIP: 0033:0x7f1a63983332 [ 2996.653143] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2996.667873] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2996.684343] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 2996.697139] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 2996.705551] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 2996.716669] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 2996.738517] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3000.056449] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 3000.060205] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3000.083770] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3000.090453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3000.096220] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3000.099069] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3000.109648] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3000.113594] RAX: 0000000000007884 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3000.117038] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3000.124753] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3000.132089] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3000.139060] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3000.145376] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3000.150023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3000.152891] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3000.156138] Call Trace: [ 3000.156933] [ 3000.157578] ? show_regs.cold.9+0x22/0x2f [ 3000.160389] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3000.163627] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3000.166132] ? watchdog+0x40/0x40 [ 3000.167816] ? hrtimer_interrupt+0x128/0x2c0 [ 3000.170128] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3000.173791] ? apic_timer_interrupt+0xf/0x20 [ 3000.177573] [ 3000.178940] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3000.189819] do_raw_spin_lock+0x177/0x190 [ 3000.192361] _raw_spin_lock+0x21/0x30 [ 3000.195540] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3000.199082] ? do_raw_spin_unlock+0x75/0x190 [ 3000.203488] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3000.206133] ? do_wait_intr+0xf0/0xf0 [ 3000.210397] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3000.214237] mdc_close+0x53b/0xd70 [mdc] [ 3000.218102] lmv_close+0x18d/0x490 [lmv] [ 3000.223234] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3000.226008] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3000.228956] ll_md_close+0x274/0x800 [lustre] [ 3000.230589] ll_file_release+0x128/0xc50 [lustre] [ 3000.233483] __fput+0x10d/0x440 [ 3000.235123] ____fput+0x12/0x20 [ 3000.236461] task_work_run+0xc6/0x110 [ 3000.238089] exit_to_usermode_loop+0x1dd/0x1f0 [ 3000.240069] do_syscall_64+0x3d6/0x440 [ 3000.241375] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3000.243390] RIP: 0033:0x7f1a63983cd8 [ 3000.244816] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3000.253396] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3000.257004] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3000.261654] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3000.264667] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3000.267796] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3000.273539] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3001.850449] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 978s! [ 3001.858333] Showing busy workqueues and worker pools: [ 3001.869394] workqueue events: flags=0x0 [ 3001.875613] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3001.883787] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3001.886402] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3001.908297] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3001.931530] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3001.942210] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3001.954124] workqueue events_long: flags=0x0 [ 3001.960014] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3001.965015] pending: lu_objects_flush [obdclass] [ 3001.969448] workqueue events_unbound: flags=0x2 [ 3001.975315] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3001.984410] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3002.005407] workqueue events_power_efficient: flags=0x80 [ 3002.008632] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3002.016014] pending: neigh_periodic_work, do_cache_clean [ 3002.023016] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3002.034526] pending: neigh_periodic_work, check_lifetime [ 3002.040373] workqueue mm_percpu_wq: flags=0x8 [ 3002.046158] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3002.055946] pending: vmstat_update [ 3002.068792] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3002.084959] pending: vmstat_update [ 3002.095920] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3002.111534] pending: vmstat_update [ 3002.114523] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3002.122526] pending: vmstat_update [ 3002.125902] workqueue writeback: flags=0x4a [ 3002.134218] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3002.148275] pending: wb_workfn, wb_workfn [ 3002.159074] workqueue xprtiod: flags=0xa [ 3002.163972] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3002.171558] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3002.179358] workqueue ipv6_addrconf: flags=0x40008 [ 3002.187364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3002.200600] pending: addrconf_verify_work [ 3002.210910] workqueue knbd0-recv: flags=0x1a [ 3002.215432] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3002.228670] in-flight: 647:recv_work [ 3002.236568] workqueue ptlrpc_pinger: flags=0x2 [ 3002.239704] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3002.248139] pending: ptlrpc_pinger_main [ptlrpc] [ 3002.259335] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2992s workers=2 idle: 652 [ 3004.057017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3004.065236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3004.138364] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3004.155058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3004.174471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3004.188434] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3004.215511] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3004.222368] RAX: 0000000000004af5 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3004.230988] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3004.243461] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3004.252897] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3004.263564] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3004.273215] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3004.290520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3004.303294] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3004.307882] Call Trace: [ 3004.309240] [ 3004.310583] ? show_regs.cold.9+0x22/0x2f [ 3004.314283] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3004.320640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3004.324301] ? watchdog+0x40/0x40 [ 3004.326413] ? hrtimer_interrupt+0x128/0x2c0 [ 3004.332178] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3004.334626] ? apic_timer_interrupt+0xf/0x20 [ 3004.338209] [ 3004.343200] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3004.348771] do_raw_spin_lock+0x177/0x190 [ 3004.351970] _raw_spin_lock+0x21/0x30 [ 3004.354721] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3004.358538] ? do_raw_spin_unlock+0x75/0x190 [ 3004.363740] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3004.366595] ? do_wait_intr+0xf0/0xf0 [ 3004.368635] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3004.370627] mdc_close+0x53b/0xd70 [mdc] [ 3004.373356] lmv_close+0x18d/0x490 [lmv] [ 3004.375249] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3004.378374] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3004.380965] ll_md_close+0x274/0x800 [lustre] [ 3004.383961] ll_file_release+0x128/0xc50 [lustre] [ 3004.393380] __fput+0x10d/0x440 [ 3004.398482] ____fput+0x12/0x20 [ 3004.400181] task_work_run+0xc6/0x110 [ 3004.407398] exit_to_usermode_loop+0x1dd/0x1f0 [ 3004.409094] do_syscall_64+0x3d6/0x440 [ 3004.419521] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3004.426668] RIP: 0033:0x7f1a63983cd8 [ 3004.437709] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3004.467312] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3004.472438] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3004.475386] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3004.480886] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3004.494901] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3004.504694] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3008.065240] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3008.071242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3008.123726] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3008.129175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3008.134877] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3008.139397] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3008.158564] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3008.166525] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3008.175690] RDX: 00000000000073d4 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3008.181498] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3008.186475] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3008.191002] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3008.202919] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3008.211189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3008.214230] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3008.217524] Call Trace: [ 3008.218892] [ 3008.219842] ? show_regs.cold.9+0x22/0x2f [ 3008.221702] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3008.224212] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3008.226255] ? watchdog+0x40/0x40 [ 3008.227657] ? hrtimer_interrupt+0x128/0x2c0 [ 3008.229432] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3008.231470] ? apic_timer_interrupt+0xf/0x20 [ 3008.233349] [ 3008.234488] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3008.237072] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3008.239342] do_raw_spin_lock+0x177/0x190 [ 3008.241292] _raw_spin_lock+0x21/0x30 [ 3008.243269] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3008.246119] obd_mod_free+0x46/0x140 [ptlrpc] [ 3008.248279] mdc_commit_open+0x91/0xb0 [mdc] [ 3008.253019] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3008.255690] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3008.264113] after_reply+0xe2c/0x1780 [ptlrpc] [ 3008.266728] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3008.277843] ? __next_timer_interrupt+0x160/0x160 [ 3008.281163] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3008.283673] ? do_wait_intr+0xf0/0xf0 [ 3008.288898] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3008.292651] kthread+0x1d1/0x200 [ 3008.294427] ? set_kthread_struct+0x70/0x70 [ 3008.297235] ret_from_fork+0x1f/0x30 [ 3028.054162] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3028.056292] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3028.056952] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 3028.077176] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3028.077189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3028.077192] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3028.077243] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3028.077247] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3028.077253] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3028.077256] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3028.077258] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3028.077260] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3028.077262] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3028.077266] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3028.077269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3028.077271] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3028.077279] Call Trace: [ 3028.077294] [ 3028.077301] ? show_regs.cold.9+0x22/0x2f [ 3028.077310] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3028.077317] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3028.077321] ? watchdog+0x40/0x40 [ 3028.077325] ? hrtimer_interrupt+0x128/0x2c0 [ 3028.077329] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3028.077336] ? apic_timer_interrupt+0xf/0x20 [ 3028.077341] [ 3028.077344] ? native_safe_halt+0x2e/0x50 [ 3028.077348] kvm_wait+0xcd/0xe0 [ 3028.077354] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3028.077363] do_raw_spin_lock+0x177/0x190 [ 3028.077368] _raw_spin_lock+0x21/0x30 [ 3028.077372] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3028.077665] ? do_raw_spin_unlock+0x75/0x190 [ 3028.077671] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3028.077884] ? do_wait_intr+0xf0/0xf0 [ 3028.077892] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3028.078105] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3028.078334] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3028.078547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3028.078806] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3028.078840] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3028.078867] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3028.078895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3028.078982] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3028.079201] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3028.079232] lmv_intent_open+0x338/0x1150 [lmv] [ 3028.079265] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3028.079356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3028.079446] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3028.079469] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3028.079667] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3028.079759] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3028.079850] ? do_raw_spin_unlock+0x75/0x190 [ 3028.079855] ? _raw_spin_unlock+0x12/0x30 [ 3028.079861] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3028.083157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3028.083237] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3028.083248] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3028.083251] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3028.083264] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3028.083268] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3028.083272] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3028.083275] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3028.083277] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3028.083279] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3028.083282] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3028.083285] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3028.083288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3028.083291] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3028.083300] Call Trace: [ 3028.083306] [ 3028.083311] ? show_regs.cold.9+0x22/0x2f [ 3028.083321] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3028.083328] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3028.083332] ? watchdog+0x40/0x40 [ 3028.083337] ? hrtimer_interrupt+0x128/0x2c0 [ 3028.083341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3028.083347] ? apic_timer_interrupt+0xf/0x20 [ 3028.083352] [ 3028.083355] ? native_safe_halt+0x2e/0x50 [ 3028.083359] kvm_wait+0xcd/0xe0 [ 3028.083366] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3028.083374] do_raw_spin_lock+0x177/0x190 [ 3028.083379] _raw_spin_lock+0x21/0x30 [ 3028.083383] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3028.083727] ? do_raw_spin_unlock+0x75/0x190 [ 3028.083733] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3028.083935] ? do_wait_intr+0xf0/0xf0 [ 3028.083942] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3028.086741] mdc_close+0x53b/0xd70 [mdc] [ 3028.086795] lmv_close+0x18d/0x490 [lmv] [ 3028.086829] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3028.086964] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3028.087047] ll_md_close+0x274/0x800 [lustre] [ 3028.087217] ll_file_release+0x128/0xc50 [lustre] [ 3028.087318] __fput+0x10d/0x440 [ 3028.087328] ____fput+0x12/0x20 [ 3028.087344] task_work_run+0xc6/0x110 [ 3028.087354] exit_to_usermode_loop+0x1dd/0x1f0 [ 3028.087361] do_syscall_64+0x3d6/0x440 [ 3028.087366] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3028.087374] RIP: 0033:0x7f1a63983cd8 [ 3028.087424] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3028.087429] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3028.087435] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3028.087437] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3028.087440] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3028.087442] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3028.087444] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3028.578797] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3028.581500] ll_atomic_open+0x36a/0xd70 [lustre] [ 3028.584334] lookup_open+0x356/0xb40 [ 3028.586193] do_last+0x313/0xfc0 [ 3028.587781] ? nd_jump_root+0xe5/0x160 [ 3028.589910] ? path_init+0x437/0x520 [ 3028.591280] path_openat+0xf7/0x500 [ 3028.592431] ? do_raw_spin_unlock+0x75/0x190 [ 3028.594260] do_filp_open+0x99/0x140 [ 3028.595815] ? getname_flags+0x6e/0x330 [ 3028.598088] ? __check_object_size+0xff/0x256 [ 3028.600552] ? do_raw_spin_unlock+0x75/0x190 [ 3028.603371] ? _raw_spin_unlock+0x12/0x30 [ 3028.605474] do_sys_openat2+0x2b4/0x410 [ 3028.607784] do_sys_open+0x73/0xa0 [ 3028.612021] __x64_sys_openat+0x24/0x30 [ 3028.614570] do_syscall_64+0xc1/0x440 [ 3028.616498] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3028.621593] RIP: 0033:0x7f1a63983332 [ 3028.625663] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3028.644203] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3028.651227] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3028.657465] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3028.671012] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3028.673553] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3028.676849] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3032.060101] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3032.077998] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3032.099414] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3032.104027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3032.107914] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3032.110304] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3032.118332] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3032.121983] RAX: 0000000000006127 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3032.125022] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3032.127739] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3032.130655] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3032.134968] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3032.138050] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3032.147489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3032.151034] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3032.157410] Call Trace: [ 3032.158914] [ 3032.159716] ? show_regs.cold.9+0x22/0x2f [ 3032.161023] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3032.162325] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3032.164442] ? watchdog+0x40/0x40 [ 3032.165933] ? hrtimer_interrupt+0x128/0x2c0 [ 3032.167959] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3032.170150] ? apic_timer_interrupt+0xf/0x20 [ 3032.172052] [ 3032.173678] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3032.179931] do_raw_spin_lock+0x177/0x190 [ 3032.181419] _raw_spin_lock+0x21/0x30 [ 3032.182392] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3032.185306] ? do_raw_spin_unlock+0x75/0x190 [ 3032.188654] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3032.191201] ? do_wait_intr+0xf0/0xf0 [ 3032.192881] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3032.194407] mdc_close+0x53b/0xd70 [mdc] [ 3032.195412] lmv_close+0x18d/0x490 [lmv] [ 3032.196448] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3032.199799] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3032.201765] ll_md_close+0x274/0x800 [lustre] [ 3032.203862] ll_file_release+0x128/0xc50 [lustre] [ 3032.211659] __fput+0x10d/0x440 [ 3032.212760] ____fput+0x12/0x20 [ 3032.213802] task_work_run+0xc6/0x110 [ 3032.218450] exit_to_usermode_loop+0x1dd/0x1f0 [ 3032.219921] do_syscall_64+0x3d6/0x440 [ 3032.220834] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3032.225025] RIP: 0033:0x7f1a63983cd8 [ 3032.226935] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3032.235061] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3032.237863] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3032.241319] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3032.244771] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3032.248158] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3032.252024] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3032.548071] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1009s! [ 3032.551152] Showing busy workqueues and worker pools: [ 3032.556289] workqueue events: flags=0x0 [ 3032.559532] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3032.566292] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3032.574792] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3032.580147] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3032.590112] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3032.592498] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3032.602112] workqueue events_long: flags=0x0 [ 3032.604256] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3032.611136] pending: lu_objects_flush [obdclass] [ 3032.614775] workqueue events_unbound: flags=0x2 [ 3032.620562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3032.622990] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3032.632046] workqueue events_power_efficient: flags=0x80 [ 3032.634778] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3032.637961] pending: neigh_periodic_work, do_cache_clean [ 3032.640910] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3032.645663] pending: neigh_periodic_work, check_lifetime [ 3032.651827] workqueue mm_percpu_wq: flags=0x8 [ 3032.654097] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3032.658123] pending: vmstat_update [ 3032.661675] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3032.664337] pending: vmstat_update [ 3032.665649] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3032.670349] pending: vmstat_update [ 3032.671269] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3032.675135] pending: vmstat_update [ 3032.678587] workqueue writeback: flags=0x4a [ 3032.681069] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3032.685617] pending: wb_workfn, wb_workfn [ 3032.688418] workqueue xprtiod: flags=0xa [ 3032.691047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3032.694446] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3032.704737] workqueue ipv6_addrconf: flags=0x40008 [ 3032.706495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3032.721538] pending: addrconf_verify_work [ 3032.725458] workqueue knbd0-recv: flags=0x1a [ 3032.728783] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3032.731499] in-flight: 647:recv_work [ 3032.734268] workqueue ptlrpc_pinger: flags=0x2 [ 3032.735793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3032.738812] pending: ptlrpc_pinger_main [ptlrpc] [ 3032.742860] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3022s workers=2 idle: 652 [ 3036.058117] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3036.064692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3036.099378] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3036.108722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3036.114089] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3036.117052] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3036.127907] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3036.131985] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3036.137983] RDX: 00000000000072b4 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3036.139880] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3036.145420] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3036.149575] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3036.154201] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3036.158910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3036.162050] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3036.165640] Call Trace: [ 3036.166965] [ 3036.168257] ? show_regs.cold.9+0x22/0x2f [ 3036.169995] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3036.172902] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3036.175043] ? watchdog+0x40/0x40 [ 3036.178114] ? hrtimer_interrupt+0x128/0x2c0 [ 3036.181256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3036.186681] ? apic_timer_interrupt+0xf/0x20 [ 3036.190746] [ 3036.191616] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3036.194089] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3036.196604] do_raw_spin_lock+0x177/0x190 [ 3036.198272] _raw_spin_lock+0x21/0x30 [ 3036.199830] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3036.202435] obd_mod_free+0x46/0x140 [ptlrpc] [ 3036.205034] mdc_commit_open+0x91/0xb0 [mdc] [ 3036.207025] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3036.209888] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3036.212373] after_reply+0xe2c/0x1780 [ptlrpc] [ 3036.214850] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3036.217844] ? __next_timer_interrupt+0x160/0x160 [ 3036.219886] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3036.221694] ? do_wait_intr+0xf0/0xf0 [ 3036.222753] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3036.224968] kthread+0x1d1/0x200 [ 3036.226277] ? set_kthread_struct+0x70/0x70 [ 3036.228065] ret_from_fork+0x1f/0x30 [ 3056.056010] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 3056.059710] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3056.083817] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3056.089350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3056.094076] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3056.096951] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3056.106370] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3056.109787] RAX: 0000000000006be2 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3056.113389] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3056.116901] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3056.120017] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3056.124055] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3056.127905] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3056.132401] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3056.135452] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3056.138627] Call Trace: [ 3056.139786] [ 3056.140911] ? show_regs.cold.9+0x22/0x2f [ 3056.142858] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3056.145148] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3056.147393] ? watchdog+0x40/0x40 [ 3056.148956] ? hrtimer_interrupt+0x128/0x2c0 [ 3056.151148] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3056.153420] ? apic_timer_interrupt+0xf/0x20 [ 3056.155618] [ 3056.156645] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3056.159776] do_raw_spin_lock+0x177/0x190 [ 3056.161714] _raw_spin_lock+0x21/0x30 [ 3056.163490] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3056.166465] ? do_raw_spin_unlock+0x75/0x190 [ 3056.168694] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3056.172302] ? do_wait_intr+0xf0/0xf0 [ 3056.173949] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3056.177044] mdc_close+0x53b/0xd70 [mdc] [ 3056.179107] lmv_close+0x18d/0x490 [lmv] [ 3056.181407] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3056.184079] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3056.186379] ll_md_close+0x274/0x800 [lustre] [ 3056.188701] ll_file_release+0x128/0xc50 [lustre] [ 3056.190927] __fput+0x10d/0x440 [ 3056.192418] ____fput+0x12/0x20 [ 3056.193803] task_work_run+0xc6/0x110 [ 3056.195407] exit_to_usermode_loop+0x1dd/0x1f0 [ 3056.197541] do_syscall_64+0x3d6/0x440 [ 3056.199153] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3056.201852] RIP: 0033:0x7f1a63983cd8 [ 3056.203554] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3056.214627] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3056.218349] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3056.221942] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3056.225702] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3056.228865] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3056.232244] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3060.053020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3060.055677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3060.057057] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3060.069402] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3060.069415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3060.069418] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3060.069430] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3060.069433] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3060.069439] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3060.069441] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3060.069443] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3060.069445] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3060.069447] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3060.069449] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3060.069452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3060.069454] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3060.069461] Call Trace: [ 3060.069465] [ 3060.069470] ? show_regs.cold.9+0x22/0x2f [ 3060.069478] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3060.069483] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3060.069487] ? watchdog+0x40/0x40 [ 3060.069491] ? hrtimer_interrupt+0x128/0x2c0 [ 3060.069494] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3060.069499] ? apic_timer_interrupt+0xf/0x20 [ 3060.069503] [ 3060.069506] ? native_safe_halt+0x2e/0x50 [ 3060.069509] kvm_wait+0xcd/0xe0 [ 3060.069515] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3060.069522] do_raw_spin_lock+0x177/0x190 [ 3060.069526] _raw_spin_lock+0x21/0x30 [ 3060.069529] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3060.069842] ? do_raw_spin_unlock+0x75/0x190 [ 3060.069846] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3060.069971] ? do_wait_intr+0xf0/0xf0 [ 3060.069977] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3060.074602] Modules linked in: [ 3060.079437] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3060.083050] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3060.083296] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3060.083303] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3060.083305] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3060.083318] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3060.083322] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3060.083332] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3060.083386] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3060.083388] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3060.083389] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3060.083391] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3060.083393] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3060.083395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3060.083397] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3060.083403] Call Trace: [ 3060.083408] [ 3060.083412] ? show_regs.cold.9+0x22/0x2f [ 3060.083420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3060.083424] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3060.083427] ? watchdog+0x40/0x40 [ 3060.083430] ? hrtimer_interrupt+0x128/0x2c0 [ 3060.083432] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3060.083437] ? apic_timer_interrupt+0xf/0x20 [ 3060.083440] [ 3060.083442] ? native_safe_halt+0x2e/0x50 [ 3060.083444] kvm_wait+0xcd/0xe0 [ 3060.083450] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3060.083455] do_raw_spin_lock+0x177/0x190 [ 3060.083458] _raw_spin_lock+0x21/0x30 [ 3060.083460] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3060.083663] ? do_raw_spin_unlock+0x75/0x190 [ 3060.083667] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3060.083823] ? do_wait_intr+0xf0/0xf0 [ 3060.083829] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3060.084442] mdc_close+0x53b/0xd70 [mdc] [ 3060.084471] lmv_close+0x18d/0x490 [lmv] [ 3060.084489] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3060.084573] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3060.084628] ll_md_close+0x274/0x800 [lustre] [ 3060.084700] ll_file_release+0x128/0xc50 [lustre] [ 3060.087004] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3060.103806] __fput+0x10d/0x440 [ 3060.103826] ____fput+0x12/0x20 [ 3060.103830] task_work_run+0xc6/0x110 [ 3060.103837] exit_to_usermode_loop+0x1dd/0x1f0 [ 3060.103843] do_syscall_64+0x3d6/0x440 [ 3060.103847] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3060.103855] RIP: 0033:0x7f1a63983cd8 [ 3060.103862] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3060.103866] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3060.103871] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3060.103873] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3060.103876] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3060.103878] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3060.103879] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3060.346106] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3060.348773] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3060.351144] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3060.353337] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3060.355828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3060.358027] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3060.360704] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3060.363115] lmv_intent_open+0x338/0x1150 [lmv] [ 3060.364738] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3060.367158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3060.369568] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3060.371773] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3060.373893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3060.376469] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3060.378554] ? do_raw_spin_unlock+0x75/0x190 [ 3060.380450] ? _raw_spin_unlock+0x12/0x30 [ 3060.382100] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3060.385627] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3060.387580] ll_atomic_open+0x36a/0xd70 [lustre] [ 3060.390346] lookup_open+0x356/0xb40 [ 3060.391474] do_last+0x313/0xfc0 [ 3060.393170] ? nd_jump_root+0xe5/0x160 [ 3060.394748] ? path_init+0x437/0x520 [ 3060.396379] path_openat+0xf7/0x500 [ 3060.401109] ? do_raw_spin_unlock+0x75/0x190 [ 3060.403106] do_filp_open+0x99/0x140 [ 3060.406969] ? getname_flags+0x6e/0x330 [ 3060.408646] ? __check_object_size+0xff/0x256 [ 3060.412460] ? do_raw_spin_unlock+0x75/0x190 [ 3060.417237] ? _raw_spin_unlock+0x12/0x30 [ 3060.418784] do_sys_openat2+0x2b4/0x410 [ 3060.420913] do_sys_open+0x73/0xa0 [ 3060.422132] __x64_sys_openat+0x24/0x30 [ 3060.423661] do_syscall_64+0xc1/0x440 [ 3060.424982] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3060.426772] RIP: 0033:0x7f1a63983332 [ 3060.428924] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3060.437923] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3060.442421] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3060.446414] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3060.451656] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3060.456847] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3060.469908] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3062.754036] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1039s! [ 3062.757121] Showing busy workqueues and worker pools: [ 3062.759542] workqueue events: flags=0x0 [ 3062.760996] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3062.764303] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3062.768018] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3062.772392] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3062.777180] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3062.780327] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3062.784739] workqueue events_long: flags=0x0 [ 3062.786671] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3062.789833] pending: lu_objects_flush [obdclass] [ 3062.792555] workqueue events_unbound: flags=0x2 [ 3062.794558] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3062.797725] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3062.802024] workqueue events_power_efficient: flags=0x80 [ 3062.804192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3062.807568] pending: neigh_periodic_work, do_cache_clean [ 3062.810328] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3062.813646] pending: neigh_periodic_work, check_lifetime [ 3062.816280] workqueue mm_percpu_wq: flags=0x8 [ 3062.818419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3062.821502] pending: vmstat_update [ 3062.823321] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3062.826416] pending: vmstat_update [ 3062.828237] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3062.831370] pending: vmstat_update [ 3062.833224] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3062.836298] pending: vmstat_update [ 3062.837398] workqueue writeback: flags=0x4a [ 3062.838860] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3062.841978] pending: wb_workfn, wb_workfn [ 3062.844243] workqueue xprtiod: flags=0xa [ 3062.846145] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3062.849027] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3062.852619] workqueue ipv6_addrconf: flags=0x40008 [ 3062.855041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3062.857694] pending: addrconf_verify_work [ 3062.859509] workqueue knbd0-recv: flags=0x1a [ 3062.861449] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3062.863818] in-flight: 647:recv_work [ 3062.865670] workqueue ptlrpc_pinger: flags=0x2 [ 3062.867521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3062.870505] pending: ptlrpc_pinger_main [ptlrpc] [ 3062.873255] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3052s workers=2 idle: 652 [ 3064.058934] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3064.066571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3064.088573] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3064.094092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3064.098547] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3064.101507] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3064.110015] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3064.121419] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3064.124893] RDX: 0000000000001bec RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3064.131886] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3064.140056] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3064.157719] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3064.180320] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3064.191298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3064.194732] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3064.197201] Call Trace: [ 3064.198190] [ 3064.205871] ? show_regs.cold.9+0x22/0x2f [ 3064.207608] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3064.210659] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3064.217237] ? watchdog+0x40/0x40 [ 3064.219432] ? hrtimer_interrupt+0x128/0x2c0 [ 3064.222530] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3064.226116] ? apic_timer_interrupt+0xf/0x20 [ 3064.229897] [ 3064.230655] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3064.238773] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3064.242055] do_raw_spin_lock+0x177/0x190 [ 3064.243976] _raw_spin_lock+0x21/0x30 [ 3064.245965] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3064.248567] obd_mod_free+0x46/0x140 [ptlrpc] [ 3064.251551] mdc_commit_open+0x91/0xb0 [mdc] [ 3064.253551] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3064.256454] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3064.258973] after_reply+0xe2c/0x1780 [ptlrpc] [ 3064.261136] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3064.263856] ? __next_timer_interrupt+0x160/0x160 [ 3064.266111] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3064.268609] ? do_wait_intr+0xf0/0xf0 [ 3064.270614] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3064.274650] kthread+0x1d1/0x200 [ 3064.276125] ? set_kthread_struct+0x70/0x70 [ 3064.278223] ret_from_fork+0x1f/0x30 [ 3084.056035] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 3084.059811] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3084.098094] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3084.111096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3084.116341] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3084.119675] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3084.136598] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3084.148519] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3084.154860] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3084.166741] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3084.174486] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3084.180427] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3084.186597] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3084.192833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3084.199600] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3084.208299] Call Trace: [ 3084.209260] [ 3084.210528] ? show_regs.cold.9+0x22/0x2f [ 3084.214497] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3084.217524] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3084.223069] ? watchdog+0x40/0x40 [ 3084.226745] ? hrtimer_interrupt+0x128/0x2c0 [ 3084.230477] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3084.244609] ? apic_timer_interrupt+0xf/0x20 [ 3084.250372] [ 3084.251549] ? native_safe_halt+0x2e/0x50 [ 3084.253532] kvm_wait+0xcd/0xe0 [ 3084.258872] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3084.264410] do_raw_spin_lock+0x177/0x190 [ 3084.266899] _raw_spin_lock+0x21/0x30 [ 3084.271846] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3084.276771] ? do_raw_spin_unlock+0x75/0x190 [ 3084.282090] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3084.286712] ? do_wait_intr+0xf0/0xf0 [ 3084.288109] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3084.292669] mdc_close+0x53b/0xd70 [mdc] [ 3084.298554] lmv_close+0x18d/0x490 [lmv] [ 3084.300813] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3084.305109] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3084.308717] ll_md_close+0x274/0x800 [lustre] [ 3084.313183] ll_file_release+0x128/0xc50 [lustre] [ 3084.316161] __fput+0x10d/0x440 [ 3084.319862] ____fput+0x12/0x20 [ 3084.321852] task_work_run+0xc6/0x110 [ 3084.324410] exit_to_usermode_loop+0x1dd/0x1f0 [ 3084.326727] do_syscall_64+0x3d6/0x440 [ 3084.329198] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3084.330985] RIP: 0033:0x7f1a63983cd8 [ 3084.339390] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3084.355696] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3084.359857] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3084.366452] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3084.372541] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3084.377192] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3084.382117] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3088.053591] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3088.057271] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3088.064537] Modules linked in: lustre(O) [ 3088.068936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 3088.073933] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 3088.096249] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3088.096293] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3088.096300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3088.096303] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3088.096317] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3088.096321] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3088.096326] RAX: 000000000000658f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3088.096329] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3088.096331] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3088.096333] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3088.096335] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3088.096339] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3088.096341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3088.096343] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3088.096352] Call Trace: [ 3088.096358] [ 3088.096364] ? show_regs.cold.9+0x22/0x2f [ 3088.096374] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3088.096381] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3088.096386] ? watchdog+0x40/0x40 [ 3088.096390] ? hrtimer_interrupt+0x128/0x2c0 [ 3088.096394] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3088.096404] ? apic_timer_interrupt+0xf/0x20 [ 3088.096408] [ 3088.096410] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3088.096417] do_raw_spin_lock+0x177/0x190 [ 3088.096421] _raw_spin_lock+0x21/0x30 [ 3088.096426] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3088.098246] ? do_raw_spin_unlock+0x75/0x190 [ 3088.098258] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3088.098503] ? do_wait_intr+0xf0/0xf0 [ 3088.098511] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3088.098724] mdc_close+0x53b/0xd70 [mdc] [ 3088.099804] lmv_close+0x18d/0x490 [lmv] [ 3088.099832] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3088.099953] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3088.100115] ll_md_close+0x274/0x800 [lustre] [ 3088.100201] ll_file_release+0x128/0xc50 [lustre] [ 3088.100283] __fput+0x10d/0x440 [ 3088.100292] ____fput+0x12/0x20 [ 3088.100297] task_work_run+0xc6/0x110 [ 3088.100304] exit_to_usermode_loop+0x1dd/0x1f0 [ 3088.100311] do_syscall_64+0x3d6/0x440 [ 3088.100316] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3088.100323] RIP: 0033:0x7f1a63983cd8 [ 3088.100331] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3088.100336] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3088.100341] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3088.100344] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3088.100346] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3088.100348] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3088.100350] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3088.476596] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3088.511940] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3088.529541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3088.537504] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3088.548284] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3088.567789] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3088.579932] RAX: 0000000000006715 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3088.595296] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3088.606175] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3088.616109] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3088.624459] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3088.634085] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3088.640546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3088.644617] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3088.654553] Call Trace: [ 3088.655509] [ 3088.656571] ? show_regs.cold.9+0x22/0x2f [ 3088.658284] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3088.661984] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3088.664450] ? watchdog+0x40/0x40 [ 3088.667495] ? hrtimer_interrupt+0x128/0x2c0 [ 3088.670265] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3088.673402] ? apic_timer_interrupt+0xf/0x20 [ 3088.677963] [ 3088.683034] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3088.692174] do_raw_spin_lock+0x177/0x190 [ 3088.696034] _raw_spin_lock+0x21/0x30 [ 3088.699681] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3088.710453] ? do_raw_spin_unlock+0x75/0x190 [ 3088.716456] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3088.727161] ? do_wait_intr+0xf0/0xf0 [ 3088.730247] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3088.733656] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3088.746570] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3088.763253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3088.766536] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3088.773258] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3088.780050] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3088.792044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3088.794162] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3088.799531] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3088.812300] lmv_intent_open+0x338/0x1150 [lmv] [ 3088.822638] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3088.839815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3088.849804] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3088.857962] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3088.865904] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3088.871480] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3088.876670] ? do_raw_spin_unlock+0x75/0x190 [ 3088.882817] ? _raw_spin_unlock+0x12/0x30 [ 3088.885462] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3088.888378] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3088.897559] ll_atomic_open+0x36a/0xd70 [lustre] [ 3088.900622] lookup_open+0x356/0xb40 [ 3088.904495] do_last+0x313/0xfc0 [ 3088.907415] ? nd_jump_root+0xe5/0x160 [ 3088.909435] ? path_init+0x437/0x520 [ 3088.914235] path_openat+0xf7/0x500 [ 3088.916489] ? do_raw_spin_unlock+0x75/0x190 [ 3088.919718] do_filp_open+0x99/0x140 [ 3088.923684] ? getname_flags+0x6e/0x330 [ 3088.925863] ? __check_object_size+0xff/0x256 [ 3088.929736] ? do_raw_spin_unlock+0x75/0x190 [ 3088.931812] ? _raw_spin_unlock+0x12/0x30 [ 3088.933416] do_sys_openat2+0x2b4/0x410 [ 3088.934884] do_sys_open+0x73/0xa0 [ 3088.939446] __x64_sys_openat+0x24/0x30 [ 3088.943946] do_syscall_64+0xc1/0x440 [ 3088.947077] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3088.952434] RIP: 0033:0x7f1a63983332 [ 3088.956088] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3088.976886] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3088.981788] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3088.987629] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3088.990951] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3089.001030] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3089.004087] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3092.058167] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3092.064518] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3092.146841] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3092.164164] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3092.179490] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3092.184687] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3092.218198] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3092.224392] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3092.233535] RDX: 00000000000060e7 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3092.250828] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3092.260745] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3092.271062] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3092.283280] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3092.291333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3092.298960] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3092.310336] Call Trace: [ 3092.311279] [ 3092.324748] ? show_regs.cold.9+0x22/0x2f [ 3092.330012] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3092.342530] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3092.345221] ? watchdog+0x40/0x40 [ 3092.355495] ? hrtimer_interrupt+0x128/0x2c0 [ 3092.357635] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3092.364244] ? apic_timer_interrupt+0xf/0x20 [ 3092.371356] [ 3092.372320] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3092.374255] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3092.382236] do_raw_spin_lock+0x177/0x190 [ 3092.383725] _raw_spin_lock+0x21/0x30 [ 3092.387118] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3092.390214] obd_mod_free+0x46/0x140 [ptlrpc] [ 3092.392722] mdc_commit_open+0x91/0xb0 [mdc] [ 3092.394310] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3092.397115] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3092.400704] after_reply+0xe2c/0x1780 [ptlrpc] [ 3092.403741] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3092.406569] ? __next_timer_interrupt+0x160/0x160 [ 3092.408588] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3092.411064] ? do_wait_intr+0xf0/0xf0 [ 3092.412373] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3092.414997] kthread+0x1d1/0x200 [ 3092.416451] ? set_kthread_struct+0x70/0x70 [ 3092.418630] ret_from_fork+0x1f/0x30 [ 3092.965425] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1069s! [ 3092.976758] Showing busy workqueues and worker pools: [ 3092.994316] workqueue events: flags=0x0 [ 3093.000593] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3093.005528] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3093.016862] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3093.034311] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3093.042305] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3093.051244] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3093.065372] workqueue events_long: flags=0x0 [ 3093.067566] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3093.080218] pending: lu_objects_flush [obdclass] [ 3093.085091] workqueue events_unbound: flags=0x2 [ 3093.089612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3093.100723] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3093.114167] workqueue events_power_efficient: flags=0x80 [ 3093.119617] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3093.127544] pending: neigh_periodic_work, do_cache_clean [ 3093.135314] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3093.143911] pending: neigh_periodic_work, check_lifetime [ 3093.151594] workqueue mm_percpu_wq: flags=0x8 [ 3093.157331] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3093.162332] pending: vmstat_update [ 3093.166169] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3093.170419] pending: vmstat_update [ 3093.174541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3093.184071] pending: vmstat_update [ 3093.191819] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3093.198154] pending: vmstat_update [ 3093.200417] workqueue writeback: flags=0x4a [ 3093.203628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3093.206059] pending: wb_workfn, wb_workfn [ 3093.208359] workqueue xprtiod: flags=0xa [ 3093.213400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3093.221473] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3093.225614] workqueue ipv6_addrconf: flags=0x40008 [ 3093.228464] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3093.234318] pending: addrconf_verify_work [ 3093.237339] workqueue knbd0-recv: flags=0x1a [ 3093.240883] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3093.243897] in-flight: 647:recv_work [ 3093.245250] workqueue ptlrpc_pinger: flags=0x2 [ 3093.246795] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3093.257455] pending: ptlrpc_pinger_main [ptlrpc] [ 3093.263360] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3083s workers=2 idle: 652 [ 3112.058746] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 3112.071449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3112.165021] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3112.186883] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3112.205658] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3112.208943] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3112.247462] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3112.269286] RAX: 0000000000001188 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3112.292438] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3112.306593] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3112.320348] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3112.339499] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3112.349345] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3112.361725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3112.373845] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3112.397743] Call Trace: [ 3112.406281] [ 3112.410787] ? show_regs.cold.9+0x22/0x2f [ 3112.419521] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3112.423945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3112.431097] ? watchdog+0x40/0x40 [ 3112.437907] ? hrtimer_interrupt+0x128/0x2c0 [ 3112.439760] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3112.449381] ? apic_timer_interrupt+0xf/0x20 [ 3112.455124] [ 3112.460452] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3112.464094] do_raw_spin_lock+0x177/0x190 [ 3112.469009] _raw_spin_lock+0x21/0x30 [ 3112.471036] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3112.477998] ? do_raw_spin_unlock+0x75/0x190 [ 3112.481099] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3112.489549] ? do_wait_intr+0xf0/0xf0 [ 3112.494694] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3112.505201] mdc_close+0x53b/0xd70 [mdc] [ 3112.508864] lmv_close+0x18d/0x490 [lmv] [ 3112.512651] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3112.520402] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3112.522570] ll_md_close+0x274/0x800 [lustre] [ 3112.536889] ll_file_release+0x128/0xc50 [lustre] [ 3112.542633] __fput+0x10d/0x440 [ 3112.549557] ____fput+0x12/0x20 [ 3112.554246] task_work_run+0xc6/0x110 [ 3112.560019] exit_to_usermode_loop+0x1dd/0x1f0 [ 3112.563096] do_syscall_64+0x3d6/0x440 [ 3112.574112] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3112.586902] RIP: 0033:0x7f1a63983cd8 [ 3112.591552] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3112.632773] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3112.648369] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3112.662556] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3112.677294] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3112.685928] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3112.697132] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3116.057803] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3116.071380] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3116.121794] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3116.127987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3116.132855] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3116.136592] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3116.147155] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3116.151243] RAX: 0000000000006618 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3116.155156] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3116.158863] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3116.162032] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3116.166566] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3116.170243] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3116.174124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3116.177065] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3116.180877] Call Trace: [ 3116.182344] [ 3116.183539] ? show_regs.cold.9+0x22/0x2f [ 3116.185544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3116.188040] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3116.190356] ? watchdog+0x40/0x40 [ 3116.191852] ? hrtimer_interrupt+0x128/0x2c0 [ 3116.194247] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3116.196744] ? apic_timer_interrupt+0xf/0x20 [ 3116.198855] [ 3116.200066] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3116.203001] do_raw_spin_lock+0x177/0x190 [ 3116.205078] _raw_spin_lock+0x21/0x30 [ 3116.206787] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3116.209475] ? do_raw_spin_unlock+0x75/0x190 [ 3116.212123] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3116.214992] ? do_wait_intr+0xf0/0xf0 [ 3116.217098] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3116.219810] mdc_close+0x53b/0xd70 [mdc] [ 3116.221933] lmv_close+0x18d/0x490 [lmv] [ 3116.223839] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3116.226875] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3116.229098] ll_md_close+0x274/0x800 [lustre] [ 3116.231424] ll_file_release+0x128/0xc50 [lustre] [ 3116.234373] __fput+0x10d/0x440 [ 3116.235599] ____fput+0x12/0x20 [ 3116.237217] task_work_run+0xc6/0x110 [ 3116.239003] exit_to_usermode_loop+0x1dd/0x1f0 [ 3116.241045] do_syscall_64+0x3d6/0x440 [ 3116.242946] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3116.245221] RIP: 0033:0x7f1a63983cd8 [ 3116.247007] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3116.256414] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3116.259886] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3116.264175] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3116.276134] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3116.291428] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3116.297743] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3120.053009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3120.058029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3120.059220] Modules linked in: [ 3120.063069] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3120.063145] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3120.063152] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3120.063155] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3120.063169] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3120.063173] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3120.063177] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3120.063180] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3120.063182] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3120.063184] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3120.063187] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3120.063190] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3120.063193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3120.063196] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3120.063204] Call Trace: [ 3120.063208] [ 3120.063214] ? show_regs.cold.9+0x22/0x2f [ 3120.063223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3120.063230] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3120.063234] ? watchdog+0x40/0x40 [ 3120.063238] ? hrtimer_interrupt+0x128/0x2c0 [ 3120.063242] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3120.063248] ? apic_timer_interrupt+0xf/0x20 [ 3120.063253] [ 3120.063256] ? native_safe_halt+0x2e/0x50 [ 3120.063260] kvm_wait+0xcd/0xe0 [ 3120.063266] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3120.063273] do_raw_spin_lock+0x177/0x190 [ 3120.063276] _raw_spin_lock+0x21/0x30 [ 3120.063281] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3120.063782] obd_mod_free+0x46/0x140 [ptlrpc] [ 3120.063978] mdc_commit_open+0x91/0xb0 [mdc] [ 3120.064071] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3120.064264] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3120.064522] after_reply+0xe2c/0x1780 [ptlrpc] [ 3120.064855] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3120.065100] ? __next_timer_interrupt+0x160/0x160 [ 3120.065110] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3120.065314] ? do_wait_intr+0xf0/0xf0 [ 3120.065324] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3120.065649] kthread+0x1d1/0x200 [ 3120.065655] ? set_kthread_struct+0x70/0x70 [ 3120.065659] ret_from_fork+0x1f/0x30 [ 3120.275197] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3120.309582] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3120.314094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3120.318396] RIP: 0010:__pv_queued_spin_lock_slowpath+0x315/0x6d0 [ 3120.321342] Code: 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 <8b> 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 eb e0 48 [ 3120.329507] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3120.333387] RAX: 0000000000005d87 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3120.336905] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3120.340506] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3120.343982] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3120.347538] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3120.350484] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3120.354821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3120.356790] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3120.360301] Call Trace: [ 3120.361477] [ 3120.362522] ? show_regs.cold.9+0x22/0x2f [ 3120.364521] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3120.366992] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3120.371546] ? watchdog+0x40/0x40 [ 3120.374575] ? hrtimer_interrupt+0x128/0x2c0 [ 3120.376469] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3120.378447] ? apic_timer_interrupt+0xf/0x20 [ 3120.380479] [ 3120.382503] ? __pv_queued_spin_lock_slowpath+0x315/0x6d0 [ 3120.385413] do_raw_spin_lock+0x177/0x190 [ 3120.387454] _raw_spin_lock+0x21/0x30 [ 3120.389505] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3120.392233] ? do_raw_spin_unlock+0x75/0x190 [ 3120.393903] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3120.396089] ? do_wait_intr+0xf0/0xf0 [ 3120.397340] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3120.399537] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3120.405257] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3120.408045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3120.410394] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3120.413040] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3120.415027] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3120.417014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3120.419202] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3120.421727] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3120.423649] lmv_intent_open+0x338/0x1150 [lmv] [ 3120.426081] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3120.428861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3120.432222] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3120.434688] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3120.436557] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3120.440371] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3120.443463] ? do_raw_spin_unlock+0x75/0x190 [ 3120.445393] ? _raw_spin_unlock+0x12/0x30 [ 3120.447798] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3120.450978] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3120.453565] ll_atomic_open+0x36a/0xd70 [lustre] [ 3120.457106] lookup_open+0x356/0xb40 [ 3120.458615] do_last+0x313/0xfc0 [ 3120.460334] ? nd_jump_root+0xe5/0x160 [ 3120.462366] ? path_init+0x437/0x520 [ 3120.464346] path_openat+0xf7/0x500 [ 3120.467848] ? do_raw_spin_unlock+0x75/0x190 [ 3120.470104] do_filp_open+0x99/0x140 [ 3120.471889] ? getname_flags+0x6e/0x330 [ 3120.473305] ? __check_object_size+0xff/0x256 [ 3120.476091] ? do_raw_spin_unlock+0x75/0x190 [ 3120.478953] ? _raw_spin_unlock+0x12/0x30 [ 3120.481919] do_sys_openat2+0x2b4/0x410 [ 3120.484868] do_sys_open+0x73/0xa0 [ 3120.487058] __x64_sys_openat+0x24/0x30 [ 3120.489755] do_syscall_64+0xc1/0x440 [ 3120.492534] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3120.495516] RIP: 0033:0x7f1a63983332 [ 3120.497815] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3120.508526] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3120.513199] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3120.516768] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3120.518565] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3120.522365] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3120.526520] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3123.683037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1100s! [ 3123.694121] Showing busy workqueues and worker pools: [ 3123.702491] workqueue events: flags=0x0 [ 3123.704784] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3123.710354] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3123.717674] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3123.721164] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3123.735479] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3123.741311] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3123.751451] workqueue events_long: flags=0x0 [ 3123.756968] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3123.769252] pending: lu_objects_flush [obdclass] [ 3123.771782] workqueue events_unbound: flags=0x2 [ 3123.775878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3123.788703] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3123.796158] workqueue events_power_efficient: flags=0x80 [ 3123.800993] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3123.805347] pending: neigh_periodic_work, do_cache_clean [ 3123.811081] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3123.816794] pending: neigh_periodic_work, check_lifetime [ 3123.824087] workqueue mm_percpu_wq: flags=0x8 [ 3123.836298] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3123.842582] pending: vmstat_update [ 3123.846085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3123.850806] pending: vmstat_update [ 3123.853601] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3123.862331] pending: vmstat_update [ 3123.865903] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3123.879739] pending: vmstat_update [ 3123.884570] workqueue writeback: flags=0x4a [ 3123.886187] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3123.894165] pending: wb_workfn, wb_workfn [ 3123.910046] workqueue xprtiod: flags=0xa [ 3123.917513] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3123.927963] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3123.936246] workqueue ipv6_addrconf: flags=0x40008 [ 3123.943494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3123.963072] pending: addrconf_verify_work [ 3123.967457] workqueue knbd0-recv: flags=0x1a [ 3123.973327] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3123.979189] in-flight: 647:recv_work [ 3123.982255] workqueue ptlrpc_pinger: flags=0x2 [ 3123.991990] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3124.003757] pending: ptlrpc_pinger_main [ptlrpc] [ 3124.008543] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3114s workers=2 idle: 652 [ 3140.056309] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3140.060643] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3140.122142] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3140.135433] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3140.153334] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3140.169794] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3140.196299] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3140.200814] RAX: 0000000000002a27 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3140.207401] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3140.210716] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3140.214103] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3140.217696] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3140.223345] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3140.235647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3140.241264] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3140.250963] Call Trace: [ 3140.251998] [ 3140.254107] ? show_regs.cold.9+0x22/0x2f [ 3140.256583] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3140.259593] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3140.262271] ? watchdog+0x40/0x40 [ 3140.264129] ? hrtimer_interrupt+0x128/0x2c0 [ 3140.266295] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3140.268675] ? apic_timer_interrupt+0xf/0x20 [ 3140.270880] [ 3140.271731] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3140.274318] do_raw_spin_lock+0x177/0x190 [ 3140.275967] _raw_spin_lock+0x21/0x30 [ 3140.277623] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3140.279748] ? do_raw_spin_unlock+0x75/0x190 [ 3140.281089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3140.283342] ? do_wait_intr+0xf0/0xf0 [ 3140.284996] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3140.287179] mdc_close+0x53b/0xd70 [mdc] [ 3140.288998] lmv_close+0x18d/0x490 [lmv] [ 3140.291127] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3140.293691] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3140.295786] ll_md_close+0x274/0x800 [lustre] [ 3140.297823] ll_file_release+0x128/0xc50 [lustre] [ 3140.300405] __fput+0x10d/0x440 [ 3140.301856] ____fput+0x12/0x20 [ 3140.303755] task_work_run+0xc6/0x110 [ 3140.306386] exit_to_usermode_loop+0x1dd/0x1f0 [ 3140.308177] do_syscall_64+0x3d6/0x440 [ 3140.312099] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3140.314898] RIP: 0033:0x7f1a63983cd8 [ 3140.316106] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3140.326758] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3140.334070] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3140.338222] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3140.341818] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3140.345836] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3140.348321] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3144.057023] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3144.060617] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3144.097075] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3144.114894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3144.134661] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3144.139028] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3144.147857] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3144.151591] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3144.154635] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3144.160847] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3144.164848] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3144.168136] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3144.185486] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3144.199780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3144.203104] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3144.212704] Call Trace: [ 3144.214321] [ 3144.215870] ? show_regs.cold.9+0x22/0x2f [ 3144.219357] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3144.225420] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3144.236913] ? watchdog+0x40/0x40 [ 3144.245538] ? hrtimer_interrupt+0x128/0x2c0 [ 3144.256212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3144.263269] ? apic_timer_interrupt+0xf/0x20 [ 3144.268787] [ 3144.272508] ? native_safe_halt+0x2e/0x50 [ 3144.276023] kvm_wait+0xcd/0xe0 [ 3144.279797] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3144.285359] do_raw_spin_lock+0x177/0x190 [ 3144.292591] _raw_spin_lock+0x21/0x30 [ 3144.298144] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3144.311576] ? do_raw_spin_unlock+0x75/0x190 [ 3144.315278] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3144.323905] ? do_wait_intr+0xf0/0xf0 [ 3144.328643] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3144.331689] mdc_close+0x53b/0xd70 [mdc] [ 3144.334004] lmv_close+0x18d/0x490 [lmv] [ 3144.335946] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3144.347678] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3144.357163] ll_md_close+0x274/0x800 [lustre] [ 3144.365322] ll_file_release+0x128/0xc50 [lustre] [ 3144.374104] __fput+0x10d/0x440 [ 3144.380613] ____fput+0x12/0x20 [ 3144.387108] task_work_run+0xc6/0x110 [ 3144.388907] exit_to_usermode_loop+0x1dd/0x1f0 [ 3144.392129] do_syscall_64+0x3d6/0x440 [ 3144.395111] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3144.401326] RIP: 0033:0x7f1a63983cd8 [ 3144.408852] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3144.425808] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3144.435660] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3144.442706] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3144.453521] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3144.463291] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3144.474809] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3148.054918] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3148.058960] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3148.063098] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3148.063175] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3148.063183] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3148.063186] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3148.063200] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3148.063204] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3148.063209] RAX: 0000000000003b0f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3148.063211] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3148.063214] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3148.063216] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3148.063218] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3148.063221] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3148.063223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3148.063225] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3148.063234] Call Trace: [ 3148.063240] [ 3148.063247] ? show_regs.cold.9+0x22/0x2f [ 3148.063256] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3148.063262] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3148.063266] ? watchdog+0x40/0x40 [ 3148.063270] ? hrtimer_interrupt+0x128/0x2c0 [ 3148.063274] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3148.063282] ? apic_timer_interrupt+0xf/0x20 [ 3148.063286] [ 3148.063289] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3148.063295] do_raw_spin_lock+0x177/0x190 [ 3148.063300] _raw_spin_lock+0x21/0x30 [ 3148.063305] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3148.063608] ? do_raw_spin_unlock+0x75/0x190 [ 3148.063615] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3148.064063] ? do_wait_intr+0xf0/0xf0 [ 3148.064072] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3148.064265] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3148.066568] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3148.067151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3148.067268] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3148.067304] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3148.067334] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3148.067361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3148.067966] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3148.068225] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3148.068258] lmv_intent_open+0x338/0x1150 [lmv] [ 3148.068290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3148.068377] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3148.068475] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3148.068497] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3148.068597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3148.068904] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3148.068987] ? do_raw_spin_unlock+0x75/0x190 [ 3148.068996] ? _raw_spin_unlock+0x12/0x30 [ 3148.069004] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3148.069036] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3148.069055] ll_atomic_open+0x36a/0xd70 [lustre] [ 3148.069141] lookup_open+0x356/0xb40 [ 3148.069150] do_last+0x313/0xfc0 [ 3148.069155] ? nd_jump_root+0xe5/0x160 [ 3148.069160] ? path_init+0x437/0x520 [ 3148.069164] path_openat+0xf7/0x500 [ 3148.069169] ? do_raw_spin_unlock+0x75/0x190 [ 3148.069173] do_filp_open+0x99/0x140 [ 3148.069179] ? getname_flags+0x6e/0x330 [ 3148.069183] ? __check_object_size+0xff/0x256 [ 3148.069190] ? do_raw_spin_unlock+0x75/0x190 [ 3148.069193] ? _raw_spin_unlock+0x12/0x30 [ 3148.069197] do_sys_openat2+0x2b4/0x410 [ 3148.069201] do_sys_open+0x73/0xa0 [ 3148.069205] __x64_sys_openat+0x24/0x30 [ 3148.069209] do_syscall_64+0xc1/0x440 [ 3148.069215] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3148.069221] RIP: 0033:0x7f1a63983332 [ 3148.069229] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3148.069233] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3148.069239] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3148.069241] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3148.069244] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3148.069246] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3148.069249] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3148.535538] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3148.602978] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3148.627031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3148.637162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3148.640301] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3148.660182] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3148.667492] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3148.676199] RDX: 0000000000005ffe RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3148.682119] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3148.697082] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3148.715024] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3148.729137] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3148.748378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3148.753021] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3148.756341] Call Trace: [ 3148.757645] [ 3148.758761] ? show_regs.cold.9+0x22/0x2f [ 3148.765723] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3148.769176] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3148.775274] ? watchdog+0x40/0x40 [ 3148.779171] ? hrtimer_interrupt+0x128/0x2c0 [ 3148.782885] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3148.788039] ? apic_timer_interrupt+0xf/0x20 [ 3148.790907] [ 3148.792250] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3148.797014] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3148.800469] do_raw_spin_lock+0x177/0x190 [ 3148.803978] _raw_spin_lock+0x21/0x30 [ 3148.810934] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3148.814520] obd_mod_free+0x46/0x140 [ptlrpc] [ 3148.818699] mdc_commit_open+0x91/0xb0 [mdc] [ 3148.821238] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3148.829575] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3148.836272] after_reply+0xe2c/0x1780 [ptlrpc] [ 3148.842179] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3148.846100] ? __next_timer_interrupt+0x160/0x160 [ 3148.848848] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3148.852102] ? do_wait_intr+0xf0/0xf0 [ 3148.856731] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3148.862509] kthread+0x1d1/0x200 [ 3148.868071] ? set_kthread_struct+0x70/0x70 [ 3148.870060] ret_from_fork+0x1f/0x30 [ 3154.400497] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1131s! [ 3154.406925] Showing busy workqueues and worker pools: [ 3154.410016] workqueue events: flags=0x0 [ 3154.412664] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3154.416805] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3154.421285] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3154.425834] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3154.432146] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3154.445592] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3154.460381] workqueue events_long: flags=0x0 [ 3154.463552] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3154.469554] pending: lu_objects_flush [obdclass] [ 3154.473231] workqueue events_unbound: flags=0x2 [ 3154.475631] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3154.479164] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3154.485594] workqueue events_power_efficient: flags=0x80 [ 3154.493243] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3154.502629] pending: neigh_periodic_work, do_cache_clean [ 3154.507239] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3154.513284] pending: neigh_periodic_work, check_lifetime [ 3154.522140] workqueue mm_percpu_wq: flags=0x8 [ 3154.527203] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3154.534747] pending: vmstat_update [ 3154.538157] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3154.543138] pending: vmstat_update [ 3154.545299] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3154.550153] pending: vmstat_update [ 3154.553307] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3154.557631] pending: vmstat_update [ 3154.567051] workqueue writeback: flags=0x4a [ 3154.570764] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3154.579894] pending: wb_workfn, wb_workfn [ 3154.586140] workqueue xprtiod: flags=0xa [ 3154.591959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3154.609315] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3154.623504] workqueue ipv6_addrconf: flags=0x40008 [ 3154.630112] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3154.637360] pending: addrconf_verify_work [ 3154.639888] workqueue knbd0-recv: flags=0x1a [ 3154.642428] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3154.646552] in-flight: 647:recv_work [ 3154.651233] workqueue ptlrpc_pinger: flags=0x2 [ 3154.655538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3154.660140] pending: ptlrpc_pinger_main [ptlrpc] [ 3154.664153] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3144s workers=2 idle: 652 [ 3162.437065] rcu: INFO: rcu_sched self-detected stall on CPU [ 3162.439381] rcu: 3-...!: (640457 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 3162.445247] (t=1140038 jiffies g=269633 q=659) [ 3162.447505] rcu: rcu_sched kthread starved for 1139684 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3162.452380] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3162.456939] rcu: RCU grace-period kthread stack dump: [ 3162.459683] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3162.464438] Call Trace: [ 3162.465412] __schedule+0x351/0xcb0 [ 3162.466720] schedule+0xc0/0x180 [ 3162.468411] schedule_timeout+0xb4/0x190 [ 3162.470511] ? do_raw_spin_unlock+0x75/0x190 [ 3162.472713] ? __next_timer_interrupt+0x160/0x160 [ 3162.474905] ? prepare_to_swait_event+0xf3/0x140 [ 3162.477377] rcu_gp_kthread+0x8dd/0x15b0 [ 3162.479575] ? force_qs_rnp+0x2c0/0x2c0 [ 3162.481190] kthread+0x1d1/0x200 [ 3162.482666] ? set_kthread_struct+0x70/0x70 [ 3162.484674] ret_from_fork+0x1f/0x30 [ 3162.486526] rcu: Stack dump where RCU GP kthread last ran: [ 3162.488677] NMI backtrace for cpu 3 [ 3162.489993] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3162.494933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3162.499206] Call Trace: [ 3162.500657] [ 3162.501720] ? dump_stack+0xbb/0x10e [ 3162.503521] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3162.505898] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3162.508199] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3162.510714] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3162.513241] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3162.515845] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 3162.518558] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 3162.520688] ? update_process_times+0x6d/0xc0 [ 3162.522893] ? tick_sched_handle.isra.18+0x27/0x90 [ 3162.525081] ? tick_sched_timer+0x53/0xb0 [ 3162.526765] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3162.528888] ? tick_sched_handle.isra.18+0x90/0x90 [ 3162.531198] ? hrtimer_interrupt+0x128/0x2c0 [ 3162.533396] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3162.535576] ? apic_timer_interrupt+0xf/0x20 [ 3162.537507] [ 3162.538347] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3162.540897] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3162.543275] ? do_raw_spin_lock+0x177/0x190 [ 3162.544872] ? _raw_spin_lock+0x21/0x30 [ 3162.546538] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3162.549136] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3162.551558] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3162.553684] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3162.556893] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3162.559995] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3162.562730] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3162.565777] ? __next_timer_interrupt+0x160/0x160 [ 3162.568083] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3162.570359] ? do_wait_intr+0xf0/0xf0 [ 3162.572436] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3162.575042] ? kthread+0x1d1/0x200 [ 3162.576648] ? set_kthread_struct+0x70/0x70 [ 3162.579170] ? ret_from_fork+0x1f/0x30 [ 3162.581111] NMI backtrace for cpu 3 [ 3162.582860] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3162.587760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3162.591768] Call Trace: [ 3162.592764] [ 3162.593619] ? dump_stack+0xbb/0x10e [ 3162.595156] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3162.597402] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3162.599012] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3162.600871] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3162.602885] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3162.605673] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3162.607629] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3162.609757] ? update_process_times+0x6d/0xc0 [ 3162.611690] ? tick_sched_handle.isra.18+0x27/0x90 [ 3162.613433] ? tick_sched_timer+0x53/0xb0 [ 3162.614947] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3162.616903] ? tick_sched_handle.isra.18+0x90/0x90 [ 3162.618705] ? hrtimer_interrupt+0x128/0x2c0 [ 3162.620604] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3162.622694] ? apic_timer_interrupt+0xf/0x20 [ 3162.624722] [ 3162.625720] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3162.627932] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3162.630486] ? do_raw_spin_lock+0x177/0x190 [ 3162.632267] ? _raw_spin_lock+0x21/0x30 [ 3162.633352] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3162.635806] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3162.638343] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3162.640558] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3162.642931] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3162.645768] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3162.648391] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3162.651002] ? __next_timer_interrupt+0x160/0x160 [ 3162.652874] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3162.655431] ? do_wait_intr+0xf0/0xf0 [ 3162.656647] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3162.659562] ? kthread+0x1d1/0x200 [ 3162.661200] ? set_kthread_struct+0x70/0x70 [ 3162.663020] ? ret_from_fork+0x1f/0x30 [ 3168.056664] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3168.064614] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3168.107690] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3168.120198] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3168.132562] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3168.135392] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3168.149623] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3168.156775] RAX: 0000000000006515 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3168.162768] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3168.166815] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3168.177683] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3168.183639] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3168.188560] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3168.196604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3168.201531] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3168.207605] Call Trace: [ 3168.208559] [ 3168.209128] ? show_regs.cold.9+0x22/0x2f [ 3168.211697] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3168.215475] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3168.217813] ? watchdog+0x40/0x40 [ 3168.220815] ? hrtimer_interrupt+0x128/0x2c0 [ 3168.223963] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3168.227868] ? apic_timer_interrupt+0xf/0x20 [ 3168.230556] [ 3168.233493] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3168.237468] do_raw_spin_lock+0x177/0x190 [ 3168.239736] _raw_spin_lock+0x21/0x30 [ 3168.243766] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3168.246805] ? do_raw_spin_unlock+0x75/0x190 [ 3168.251453] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3168.258040] ? do_wait_intr+0xf0/0xf0 [ 3168.259809] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3168.263699] mdc_close+0x53b/0xd70 [mdc] [ 3168.267671] lmv_close+0x18d/0x490 [lmv] [ 3168.269844] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3168.273946] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3168.282490] ll_md_close+0x274/0x800 [lustre] [ 3168.285676] ll_file_release+0x128/0xc50 [lustre] [ 3168.290358] __fput+0x10d/0x440 [ 3168.292901] ____fput+0x12/0x20 [ 3168.299730] task_work_run+0xc6/0x110 [ 3168.302625] exit_to_usermode_loop+0x1dd/0x1f0 [ 3168.309031] do_syscall_64+0x3d6/0x440 [ 3168.313883] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3168.324357] RIP: 0033:0x7f1a63983cd8 [ 3168.328800] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3168.361552] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3168.374715] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3168.384794] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3168.395980] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3168.400793] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3168.415209] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3172.060608] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3172.070626] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3172.124052] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3172.130129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3172.138340] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3172.142347] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3172.155492] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3172.160049] RAX: 0000000000007288 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3172.164338] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3172.167853] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3172.171864] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3172.178277] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3172.182480] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3172.187338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3172.190506] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3172.194646] Call Trace: [ 3172.197217] [ 3172.198083] ? show_regs.cold.9+0x22/0x2f [ 3172.201271] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3172.204290] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3172.207274] ? watchdog+0x40/0x40 [ 3172.209046] ? hrtimer_interrupt+0x128/0x2c0 [ 3172.212912] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3172.216301] ? apic_timer_interrupt+0xf/0x20 [ 3172.219302] [ 3172.220359] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3172.223654] do_raw_spin_lock+0x177/0x190 [ 3172.226255] _raw_spin_lock+0x21/0x30 [ 3172.228252] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3172.231728] ? do_raw_spin_unlock+0x75/0x190 [ 3172.234252] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3172.238259] ? do_wait_intr+0xf0/0xf0 [ 3172.241043] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3172.244697] mdc_close+0x53b/0xd70 [mdc] [ 3172.248403] lmv_close+0x18d/0x490 [lmv] [ 3172.250989] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3172.254405] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3172.256983] ll_md_close+0x274/0x800 [lustre] [ 3172.259742] ll_file_release+0x128/0xc50 [lustre] [ 3172.265324] __fput+0x10d/0x440 [ 3172.267255] ____fput+0x12/0x20 [ 3172.269058] task_work_run+0xc6/0x110 [ 3172.270907] exit_to_usermode_loop+0x1dd/0x1f0 [ 3172.273401] do_syscall_64+0x3d6/0x440 [ 3172.275989] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3172.279040] RIP: 0033:0x7f1a63983cd8 [ 3172.281513] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3172.304628] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3172.307192] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3172.314187] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3172.320299] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3172.328695] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3172.333896] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3180.053512] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3180.062341] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3180.119457] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3180.135461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3180.138630] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3180.149659] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3180.165962] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3180.169265] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3180.173396] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3180.177279] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3180.180869] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3180.184331] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3180.189581] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3180.196716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3180.198717] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3180.204884] Call Trace: [ 3180.209459] [ 3180.210431] ? show_regs.cold.9+0x22/0x2f [ 3180.214886] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3180.220798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3180.223829] ? watchdog+0x40/0x40 [ 3180.225857] ? hrtimer_interrupt+0x128/0x2c0 [ 3180.233254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3180.235929] ? apic_timer_interrupt+0xf/0x20 [ 3180.240466] [ 3180.245846] ? native_safe_halt+0x2e/0x50 [ 3180.247852] kvm_wait+0xcd/0xe0 [ 3180.249364] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3180.255918] do_raw_spin_lock+0x177/0x190 [ 3180.258401] _raw_spin_lock+0x21/0x30 [ 3180.259816] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3180.263363] ? do_raw_spin_unlock+0x75/0x190 [ 3180.264943] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3180.266926] ? do_wait_intr+0xf0/0xf0 [ 3180.268557] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3180.271005] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3180.273424] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3180.275999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3180.278428] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3180.280985] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3180.283247] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3180.286828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3180.295698] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3180.298512] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3180.300852] lmv_intent_open+0x338/0x1150 [lmv] [ 3180.305092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3180.308618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3180.315031] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3180.320138] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3180.330835] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3180.335589] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3180.339194] ? do_raw_spin_unlock+0x75/0x190 [ 3180.341769] ? _raw_spin_unlock+0x12/0x30 [ 3180.343750] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3180.346727] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3180.349377] ll_atomic_open+0x36a/0xd70 [lustre] [ 3180.352261] lookup_open+0x356/0xb40 [ 3180.354356] do_last+0x313/0xfc0 [ 3180.355798] ? nd_jump_root+0xe5/0x160 [ 3180.357663] ? path_init+0x437/0x520 [ 3180.360932] path_openat+0xf7/0x500 [ 3180.362257] ? do_raw_spin_unlock+0x75/0x190 [ 3180.372727] do_filp_open+0x99/0x140 [ 3180.374389] ? getname_flags+0x6e/0x330 [ 3180.376774] ? __check_object_size+0xff/0x256 [ 3180.380127] ? do_raw_spin_unlock+0x75/0x190 [ 3180.382862] ? _raw_spin_unlock+0x12/0x30 [ 3180.386674] do_sys_openat2+0x2b4/0x410 [ 3180.391317] do_sys_open+0x73/0xa0 [ 3180.393514] __x64_sys_openat+0x24/0x30 [ 3180.396206] do_syscall_64+0xc1/0x440 [ 3180.397540] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3180.404510] RIP: 0033:0x7f1a63983332 [ 3180.407251] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3180.421791] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3180.428020] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3180.440869] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3180.447141] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3180.453516] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3180.457793] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3185.126653] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1162s! [ 3185.134057] Showing busy workqueues and worker pools: [ 3185.136397] workqueue events: flags=0x0 [ 3185.140870] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3185.152741] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3185.158358] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3185.169181] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3185.173802] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3185.177405] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3185.181900] workqueue events_long: flags=0x0 [ 3185.183612] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3185.186839] pending: lu_objects_flush [obdclass] [ 3185.189552] workqueue events_unbound: flags=0x2 [ 3185.191718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3185.194812] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3185.203313] workqueue events_power_efficient: flags=0x80 [ 3185.216769] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3185.223697] pending: neigh_periodic_work, do_cache_clean [ 3185.229097] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3185.236000] pending: neigh_periodic_work, check_lifetime [ 3185.241279] workqueue mm_percpu_wq: flags=0x8 [ 3185.247270] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3185.254937] pending: vmstat_update [ 3185.258643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3185.268197] pending: vmstat_update [ 3185.269450] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3185.274309] pending: vmstat_update [ 3185.275659] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3185.280006] pending: vmstat_update [ 3185.284107] workqueue writeback: flags=0x4a [ 3185.285693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3185.290640] pending: wb_workfn, wb_workfn [ 3185.295120] workqueue xprtiod: flags=0xa [ 3185.297120] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3185.300514] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3185.304665] workqueue ipv6_addrconf: flags=0x40008 [ 3185.307597] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3185.311265] pending: addrconf_verify_work [ 3185.312966] workqueue knbd0-recv: flags=0x1a [ 3185.314109] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3185.315921] in-flight: 647:recv_work [ 3185.317700] workqueue ptlrpc_pinger: flags=0x2 [ 3185.319276] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3185.321437] pending: ptlrpc_pinger_main [ptlrpc] [ 3185.324726] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3175s workers=2 idle: 652 [ 3188.058018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 3188.062697] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3188.107225] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3188.121380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3188.128869] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3188.138640] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3188.155731] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3188.163830] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3188.170599] RDX: 0000000000003f92 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3188.176708] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3188.181542] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3188.185659] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3188.189580] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3188.197634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3188.204692] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3188.206877] Call Trace: [ 3188.207794] [ 3188.209470] ? show_regs.cold.9+0x22/0x2f [ 3188.213165] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3188.216284] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3188.218431] ? watchdog+0x40/0x40 [ 3188.223801] ? hrtimer_interrupt+0x128/0x2c0 [ 3188.235460] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3188.237194] ? apic_timer_interrupt+0xf/0x20 [ 3188.246429] [ 3188.251819] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3188.259049] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3188.262177] do_raw_spin_lock+0x177/0x190 [ 3188.263735] _raw_spin_lock+0x21/0x30 [ 3188.264984] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3188.270983] obd_mod_free+0x46/0x140 [ptlrpc] [ 3188.276410] mdc_commit_open+0x91/0xb0 [mdc] [ 3188.282295] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3188.290587] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3188.298737] after_reply+0xe2c/0x1780 [ptlrpc] [ 3188.306298] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3188.308005] ? __next_timer_interrupt+0x160/0x160 [ 3188.312643] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3188.315706] ? do_wait_intr+0xf0/0xf0 [ 3188.317065] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3188.318957] kthread+0x1d1/0x200 [ 3188.332553] ? set_kthread_struct+0x70/0x70 [ 3188.337542] ret_from_fork+0x1f/0x30 [ 3196.056813] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3196.061546] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3196.080421] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3196.085408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3196.088492] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3196.090316] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3196.095671] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3196.097801] RAX: 0000000000000513 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3196.099754] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3196.101708] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3196.103396] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3196.105236] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3196.107164] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3196.109302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3196.111065] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3196.113114] Call Trace: [ 3196.114109] [ 3196.114721] ? show_regs.cold.9+0x22/0x2f [ 3196.115840] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3196.117428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3196.118891] ? watchdog+0x40/0x40 [ 3196.120298] ? hrtimer_interrupt+0x128/0x2c0 [ 3196.121885] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3196.123786] ? apic_timer_interrupt+0xf/0x20 [ 3196.125357] [ 3196.126296] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3196.127913] do_raw_spin_lock+0x177/0x190 [ 3196.129066] _raw_spin_lock+0x21/0x30 [ 3196.129965] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3196.131585] ? do_raw_spin_unlock+0x75/0x190 [ 3196.132734] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3196.134211] ? do_wait_intr+0xf0/0xf0 [ 3196.135214] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3196.137149] mdc_close+0x53b/0xd70 [mdc] [ 3196.138258] lmv_close+0x18d/0x490 [lmv] [ 3196.139345] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3196.140975] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3196.142308] ll_md_close+0x274/0x800 [lustre] [ 3196.143516] ll_file_release+0x128/0xc50 [lustre] [ 3196.144807] __fput+0x10d/0x440 [ 3196.145795] ____fput+0x12/0x20 [ 3196.146880] task_work_run+0xc6/0x110 [ 3196.148109] exit_to_usermode_loop+0x1dd/0x1f0 [ 3196.154859] do_syscall_64+0x3d6/0x440 [ 3196.156734] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3196.159124] RIP: 0033:0x7f1a63983cd8 [ 3196.161401] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3196.170542] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3196.173970] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3196.176879] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3196.180186] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3196.185602] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3196.190759] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3200.059927] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3200.066376] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3200.089384] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3200.097059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3200.101414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3200.102979] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3200.110095] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3200.112890] RAX: 0000000000003908 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3200.126389] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3200.132691] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3200.138307] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3200.151698] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3200.159351] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3200.163356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3200.165452] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3200.169629] Call Trace: [ 3200.171204] [ 3200.172425] ? show_regs.cold.9+0x22/0x2f [ 3200.175334] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3200.177353] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3200.179681] ? watchdog+0x40/0x40 [ 3200.182522] ? hrtimer_interrupt+0x128/0x2c0 [ 3200.192592] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3200.200181] ? apic_timer_interrupt+0xf/0x20 [ 3200.203280] [ 3200.204351] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3200.208509] do_raw_spin_lock+0x177/0x190 [ 3200.212110] _raw_spin_lock+0x21/0x30 [ 3200.214142] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3200.216062] ? do_raw_spin_unlock+0x75/0x190 [ 3200.220303] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3200.223300] ? do_wait_intr+0xf0/0xf0 [ 3200.224958] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3200.227115] mdc_close+0x53b/0xd70 [mdc] [ 3200.231051] lmv_close+0x18d/0x490 [lmv] [ 3200.234020] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3200.244278] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3200.251246] ll_md_close+0x274/0x800 [lustre] [ 3200.254961] ll_file_release+0x128/0xc50 [lustre] [ 3200.257780] __fput+0x10d/0x440 [ 3200.260238] ____fput+0x12/0x20 [ 3200.262537] task_work_run+0xc6/0x110 [ 3200.264766] exit_to_usermode_loop+0x1dd/0x1f0 [ 3200.267847] do_syscall_64+0x3d6/0x440 [ 3200.269294] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3200.272289] RIP: 0033:0x7f1a63983cd8 [ 3200.274160] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3200.282075] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3200.285621] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3200.289063] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3200.297605] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3200.305814] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3200.311130] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3212.053051] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3212.058595] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3212.087684] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3212.092757] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3212.102118] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3212.106564] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3212.120627] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3212.125080] RAX: 0000000000003e1f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3212.130663] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3212.135096] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3212.139619] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3212.143852] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3212.147966] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3212.152567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3212.157595] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3212.162993] Call Trace: [ 3212.164475] [ 3212.166330] ? show_regs.cold.9+0x22/0x2f [ 3212.168386] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3212.171236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3212.173995] ? watchdog+0x40/0x40 [ 3212.175753] ? hrtimer_interrupt+0x128/0x2c0 [ 3212.179015] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3212.185180] ? apic_timer_interrupt+0xf/0x20 [ 3212.188785] [ 3212.191688] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3212.197061] do_raw_spin_lock+0x177/0x190 [ 3212.203355] _raw_spin_lock+0x21/0x30 [ 3212.207129] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3212.213900] ? do_raw_spin_unlock+0x75/0x190 [ 3212.221985] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3212.229943] ? do_wait_intr+0xf0/0xf0 [ 3212.233307] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3212.240704] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3212.250011] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3212.256563] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3212.258997] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3212.260589] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3212.265420] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3212.268108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3212.271440] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3212.283653] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3212.286525] lmv_intent_open+0x338/0x1150 [lmv] [ 3212.295947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3212.305094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3212.312975] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3212.317981] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3212.319909] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3212.324730] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3212.332443] ? do_raw_spin_unlock+0x75/0x190 [ 3212.337444] ? _raw_spin_unlock+0x12/0x30 [ 3212.340268] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3212.343537] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3212.346913] ll_atomic_open+0x36a/0xd70 [lustre] [ 3212.350452] lookup_open+0x356/0xb40 [ 3212.354277] do_last+0x313/0xfc0 [ 3212.355483] ? nd_jump_root+0xe5/0x160 [ 3212.357488] ? path_init+0x437/0x520 [ 3212.360144] path_openat+0xf7/0x500 [ 3212.364470] ? do_raw_spin_unlock+0x75/0x190 [ 3212.367426] do_filp_open+0x99/0x140 [ 3212.372552] ? getname_flags+0x6e/0x330 [ 3212.378022] ? __check_object_size+0xff/0x256 [ 3212.385054] ? do_raw_spin_unlock+0x75/0x190 [ 3212.389690] ? _raw_spin_unlock+0x12/0x30 [ 3212.396068] do_sys_openat2+0x2b4/0x410 [ 3212.398981] do_sys_open+0x73/0xa0 [ 3212.401578] __x64_sys_openat+0x24/0x30 [ 3212.408016] do_syscall_64+0xc1/0x440 [ 3212.409381] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3212.414738] RIP: 0033:0x7f1a63983332 [ 3212.415729] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3212.427473] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3212.430510] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3212.437036] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3212.441553] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3212.445475] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3212.449067] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3215.331981] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1192s! [ 3215.338666] Showing busy workqueues and worker pools: [ 3215.345937] workqueue events: flags=0x0 [ 3215.348264] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3215.352023] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3215.360942] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3215.366531] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3215.382195] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3215.385492] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3215.400820] workqueue events_long: flags=0x0 [ 3215.404719] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3215.410132] pending: lu_objects_flush [obdclass] [ 3215.416217] workqueue events_unbound: flags=0x2 [ 3215.420493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3215.429091] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3215.440390] workqueue events_power_efficient: flags=0x80 [ 3215.444985] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3215.452093] pending: neigh_periodic_work, do_cache_clean [ 3215.456271] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3215.461984] pending: neigh_periodic_work, check_lifetime [ 3215.465494] workqueue mm_percpu_wq: flags=0x8 [ 3215.472972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3215.476952] pending: vmstat_update [ 3215.478441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3215.490722] pending: vmstat_update [ 3215.495546] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3215.499938] pending: vmstat_update [ 3215.502290] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3215.513432] pending: vmstat_update [ 3215.518423] workqueue writeback: flags=0x4a [ 3215.523233] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3215.530432] pending: wb_workfn, wb_workfn [ 3215.534647] workqueue xprtiod: flags=0xa [ 3215.537254] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3215.542113] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3215.551504] workqueue ipv6_addrconf: flags=0x40008 [ 3215.558875] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3215.567132] pending: addrconf_verify_work [ 3215.572436] workqueue knbd0-recv: flags=0x1a [ 3215.577372] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3215.583523] in-flight: 647:recv_work [ 3215.585315] workqueue ptlrpc_pinger: flags=0x2 [ 3215.591424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3215.598422] pending: ptlrpc_pinger_main [ptlrpc] [ 3215.604608] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3205s workers=2 idle: 652 [ 3216.058007] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 3216.061656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3216.082147] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3216.087018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3216.090801] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3216.093908] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3216.103324] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3216.107123] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3216.110600] RDX: 0000000000000b8f RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3216.113957] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3216.117034] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3216.120045] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3216.123050] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3216.126595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3216.129605] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3216.133083] Call Trace: [ 3216.134396] [ 3216.135413] ? show_regs.cold.9+0x22/0x2f [ 3216.137306] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3216.139411] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3216.141508] ? watchdog+0x40/0x40 [ 3216.143177] ? hrtimer_interrupt+0x128/0x2c0 [ 3216.145101] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3216.147155] ? apic_timer_interrupt+0xf/0x20 [ 3216.149186] [ 3216.150207] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3216.152303] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3216.154629] do_raw_spin_lock+0x177/0x190 [ 3216.156259] _raw_spin_lock+0x21/0x30 [ 3216.157956] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3216.160674] obd_mod_free+0x46/0x140 [ptlrpc] [ 3216.162904] mdc_commit_open+0x91/0xb0 [mdc] [ 3216.165083] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3216.167142] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3216.169417] after_reply+0xe2c/0x1780 [ptlrpc] [ 3216.171887] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3216.174256] ? __next_timer_interrupt+0x160/0x160 [ 3216.176569] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3216.179142] ? do_wait_intr+0xf0/0xf0 [ 3216.180883] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3216.184116] kthread+0x1d1/0x200 [ 3216.185754] ? set_kthread_struct+0x70/0x70 [ 3216.187693] ret_from_fork+0x1f/0x30 [ 3224.056701] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3224.067907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3224.114105] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3224.123006] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3224.130050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3224.134090] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3224.147611] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3224.154030] RAX: 000000000000523e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3224.158225] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3224.161900] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3224.165803] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3224.170001] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3224.177944] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3224.180488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3224.185757] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3224.194171] Call Trace: [ 3224.196357] [ 3224.198306] ? show_regs.cold.9+0x22/0x2f [ 3224.204290] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3224.211159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3224.216775] ? watchdog+0x40/0x40 [ 3224.219141] ? hrtimer_interrupt+0x128/0x2c0 [ 3224.221307] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3224.224977] ? apic_timer_interrupt+0xf/0x20 [ 3224.226922] [ 3224.230022] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3224.235874] do_raw_spin_lock+0x177/0x190 [ 3224.239881] _raw_spin_lock+0x21/0x30 [ 3224.244971] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3224.248075] ? do_raw_spin_unlock+0x75/0x190 [ 3224.251278] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3224.254313] ? do_wait_intr+0xf0/0xf0 [ 3224.256284] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3224.264201] mdc_close+0x53b/0xd70 [mdc] [ 3224.275153] lmv_close+0x18d/0x490 [lmv] [ 3224.279155] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3224.283211] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3224.286129] ll_md_close+0x274/0x800 [lustre] [ 3224.293406] ll_file_release+0x128/0xc50 [lustre] [ 3224.296231] __fput+0x10d/0x440 [ 3224.298091] ____fput+0x12/0x20 [ 3224.300290] task_work_run+0xc6/0x110 [ 3224.302816] exit_to_usermode_loop+0x1dd/0x1f0 [ 3224.305996] do_syscall_64+0x3d6/0x440 [ 3224.309790] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3224.314379] RIP: 0033:0x7f1a63983cd8 [ 3224.316093] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3224.325644] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3224.330843] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3224.336939] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3224.343858] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3224.346064] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3224.349315] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3228.057008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3228.062794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3228.088709] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3228.093772] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3228.096711] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3228.098428] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3228.105840] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3228.108130] RAX: 0000000000004643 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3228.110493] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3228.113800] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3228.117065] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3228.119287] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3228.122211] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3228.125141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3228.127072] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3228.129536] Call Trace: [ 3228.130378] [ 3228.130959] ? show_regs.cold.9+0x22/0x2f [ 3228.132088] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3228.134049] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3228.135597] ? watchdog+0x40/0x40 [ 3228.136909] ? hrtimer_interrupt+0x128/0x2c0 [ 3228.138520] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3228.140112] ? apic_timer_interrupt+0xf/0x20 [ 3228.141747] [ 3228.142472] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3228.144062] do_raw_spin_lock+0x177/0x190 [ 3228.145244] _raw_spin_lock+0x21/0x30 [ 3228.146190] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3228.148126] ? do_raw_spin_unlock+0x75/0x190 [ 3228.150630] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3228.153760] ? do_wait_intr+0xf0/0xf0 [ 3228.156547] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3228.159386] mdc_close+0x53b/0xd70 [mdc] [ 3228.163045] lmv_close+0x18d/0x490 [lmv] [ 3228.165131] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3228.168088] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3228.169929] ll_md_close+0x274/0x800 [lustre] [ 3228.172696] ll_file_release+0x128/0xc50 [lustre] [ 3228.174396] __fput+0x10d/0x440 [ 3228.175445] ____fput+0x12/0x20 [ 3228.176760] task_work_run+0xc6/0x110 [ 3228.177867] exit_to_usermode_loop+0x1dd/0x1f0 [ 3228.179256] do_syscall_64+0x3d6/0x440 [ 3228.180590] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3228.182255] RIP: 0033:0x7f1a63983cd8 [ 3228.183578] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3228.189895] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3228.195382] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3228.199024] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3228.201815] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3228.204549] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3228.206943] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3240.055066] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3240.062998] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3240.105886] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3240.111907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3240.117161] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3240.124867] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3240.142697] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3240.154212] RAX: 0000000000007b1f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3240.156446] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3240.162278] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3240.170615] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3240.178211] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3240.185598] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3240.193369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3240.200458] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3240.211067] Call Trace: [ 3240.213091] [ 3240.216733] ? show_regs.cold.9+0x22/0x2f [ 3240.218042] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3240.225318] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3240.230493] ? watchdog+0x40/0x40 [ 3240.232439] ? hrtimer_interrupt+0x128/0x2c0 [ 3240.234883] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3240.241283] ? apic_timer_interrupt+0xf/0x20 [ 3240.244670] [ 3240.246335] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3240.252122] do_raw_spin_lock+0x177/0x190 [ 3240.254048] _raw_spin_lock+0x21/0x30 [ 3240.257804] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3240.262094] ? do_raw_spin_unlock+0x75/0x190 [ 3240.263672] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3240.271057] ? do_wait_intr+0xf0/0xf0 [ 3240.274865] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3240.279775] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3240.286180] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3240.295276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3240.301215] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3240.305119] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3240.309268] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3240.313561] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3240.315403] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3240.327960] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3240.336923] lmv_intent_open+0x338/0x1150 [lmv] [ 3240.340290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3240.356513] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3240.358637] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3240.366120] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3240.367993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3240.376137] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3240.380066] ? do_raw_spin_unlock+0x75/0x190 [ 3240.387246] ? _raw_spin_unlock+0x12/0x30 [ 3240.391297] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3240.397623] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3240.401742] ll_atomic_open+0x36a/0xd70 [lustre] [ 3240.407679] lookup_open+0x356/0xb40 [ 3240.412923] do_last+0x313/0xfc0 [ 3240.415376] ? nd_jump_root+0xe5/0x160 [ 3240.418819] ? path_init+0x437/0x520 [ 3240.420183] path_openat+0xf7/0x500 [ 3240.433974] ? do_raw_spin_unlock+0x75/0x190 [ 3240.435814] do_filp_open+0x99/0x140 [ 3240.436855] ? getname_flags+0x6e/0x330 [ 3240.445214] ? __check_object_size+0xff/0x256 [ 3240.450089] ? do_raw_spin_unlock+0x75/0x190 [ 3240.454170] ? _raw_spin_unlock+0x12/0x30 [ 3240.462609] do_sys_openat2+0x2b4/0x410 [ 3240.467062] do_sys_open+0x73/0xa0 [ 3240.470906] __x64_sys_openat+0x24/0x30 [ 3240.474969] do_syscall_64+0xc1/0x440 [ 3240.476416] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3240.485766] RIP: 0033:0x7f1a63983332 [ 3240.487861] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3240.501441] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3240.508189] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3240.518664] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3240.524301] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3240.529691] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3240.545145] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3244.058006] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 3244.060272] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3244.100658] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3244.108105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3244.113319] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3244.117050] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3244.138450] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3244.141350] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3244.143886] RDX: 00000000000001ad RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3244.147006] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3244.156836] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3244.159976] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3244.165317] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3244.167910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3244.170069] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3244.172603] Call Trace: [ 3244.174673] [ 3244.177077] ? show_regs.cold.9+0x22/0x2f [ 3244.179252] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3244.183930] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3244.186060] ? watchdog+0x40/0x40 [ 3244.189318] ? hrtimer_interrupt+0x128/0x2c0 [ 3244.191083] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3244.193045] ? apic_timer_interrupt+0xf/0x20 [ 3244.195809] [ 3244.198435] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3244.200947] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3244.203100] do_raw_spin_lock+0x177/0x190 [ 3244.205417] _raw_spin_lock+0x21/0x30 [ 3244.207351] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3244.208995] obd_mod_free+0x46/0x140 [ptlrpc] [ 3244.211022] mdc_commit_open+0x91/0xb0 [mdc] [ 3244.212776] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3244.214874] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3244.217269] after_reply+0xe2c/0x1780 [ptlrpc] [ 3244.219008] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3244.223112] ? __next_timer_interrupt+0x160/0x160 [ 3244.224795] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3244.227570] ? do_wait_intr+0xf0/0xf0 [ 3244.229868] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3244.231726] kthread+0x1d1/0x200 [ 3244.233642] ? set_kthread_struct+0x70/0x70 [ 3244.234925] ret_from_fork+0x1f/0x30 [ 3246.048649] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1223s! [ 3246.058427] Showing busy workqueues and worker pools: [ 3246.061783] workqueue events: flags=0x0 [ 3246.063257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3246.065846] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3246.068723] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3246.071308] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3246.075346] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3246.084581] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3246.095236] workqueue events_long: flags=0x0 [ 3246.096750] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3246.101969] pending: lu_objects_flush [obdclass] [ 3246.104178] workqueue events_unbound: flags=0x2 [ 3246.106084] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3246.108726] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3246.115915] workqueue events_power_efficient: flags=0x80 [ 3246.122182] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3246.132656] pending: neigh_periodic_work, do_cache_clean [ 3246.140591] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3246.149368] pending: neigh_periodic_work, check_lifetime [ 3246.152124] workqueue mm_percpu_wq: flags=0x8 [ 3246.154546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3246.158175] pending: vmstat_update [ 3246.159823] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3246.165553] pending: vmstat_update [ 3246.170097] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3246.174108] pending: vmstat_update [ 3246.178186] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3246.186984] pending: vmstat_update [ 3246.188337] workqueue writeback: flags=0x4a [ 3246.195290] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3246.200885] pending: wb_workfn, wb_workfn [ 3246.209355] workqueue xprtiod: flags=0xa [ 3246.212602] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3246.215677] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3246.220517] workqueue ipv6_addrconf: flags=0x40008 [ 3246.222829] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3246.233418] pending: addrconf_verify_work [ 3246.240301] workqueue knbd0-recv: flags=0x1a [ 3246.247423] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3246.255229] in-flight: 647:recv_work [ 3246.263495] workqueue ptlrpc_pinger: flags=0x2 [ 3246.270817] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3246.274395] pending: ptlrpc_pinger_main [ptlrpc] [ 3246.277501] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3236s workers=2 idle: 652 [ 3252.057929] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3252.065327] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3252.103575] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3252.109562] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3252.114835] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3252.116478] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3252.137947] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3252.144687] RAX: 0000000000004041 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3252.153231] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3252.162311] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3252.170777] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3252.180766] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3252.187879] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3252.192605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3252.194970] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3252.201877] Call Trace: [ 3252.202743] [ 3252.203557] ? show_regs.cold.9+0x22/0x2f [ 3252.207102] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3252.209641] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3252.212127] ? watchdog+0x40/0x40 [ 3252.218424] ? hrtimer_interrupt+0x128/0x2c0 [ 3252.219826] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3252.231447] ? apic_timer_interrupt+0xf/0x20 [ 3252.237369] [ 3252.241027] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3252.244590] do_raw_spin_lock+0x177/0x190 [ 3252.247476] _raw_spin_lock+0x21/0x30 [ 3252.250740] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3252.262176] ? do_raw_spin_unlock+0x75/0x190 [ 3252.264830] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3252.267775] ? do_wait_intr+0xf0/0xf0 [ 3252.270912] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3252.275899] mdc_close+0x53b/0xd70 [mdc] [ 3252.278490] lmv_close+0x18d/0x490 [lmv] [ 3252.279963] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3252.286592] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3252.289659] ll_md_close+0x274/0x800 [lustre] [ 3252.293244] ll_file_release+0x128/0xc50 [lustre] [ 3252.295964] __fput+0x10d/0x440 [ 3252.298013] ____fput+0x12/0x20 [ 3252.300001] task_work_run+0xc6/0x110 [ 3252.308626] exit_to_usermode_loop+0x1dd/0x1f0 [ 3252.311780] do_syscall_64+0x3d6/0x440 [ 3252.313600] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3252.315448] RIP: 0033:0x7f1a63983cd8 [ 3252.316648] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3252.324478] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3252.330523] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3252.337308] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3252.339079] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3252.345894] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3252.349759] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3256.057020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3256.058945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3256.071687] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3256.074580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3256.077047] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3256.078205] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3256.084135] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3256.086709] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3256.089844] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3256.092009] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3256.094412] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3256.096564] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3256.098493] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3256.101073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3256.103067] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3256.105323] Call Trace: [ 3256.106231] [ 3256.106946] ? show_regs.cold.9+0x22/0x2f [ 3256.108135] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3256.109720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3256.110938] ? watchdog+0x40/0x40 [ 3256.111863] ? hrtimer_interrupt+0x128/0x2c0 [ 3256.112987] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3256.114328] ? apic_timer_interrupt+0xf/0x20 [ 3256.115570] [ 3256.116236] ? native_safe_halt+0x2e/0x50 [ 3256.117591] kvm_wait+0xcd/0xe0 [ 3256.119127] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3256.121243] do_raw_spin_lock+0x177/0x190 [ 3256.123968] _raw_spin_lock+0x21/0x30 [ 3256.126031] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3256.131883] ? do_raw_spin_unlock+0x75/0x190 [ 3256.136074] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3256.145171] ? do_wait_intr+0xf0/0xf0 [ 3256.146684] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3256.150971] mdc_close+0x53b/0xd70 [mdc] [ 3256.154768] lmv_close+0x18d/0x490 [lmv] [ 3256.156207] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3256.158184] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3256.160025] ll_md_close+0x274/0x800 [lustre] [ 3256.161915] ll_file_release+0x128/0xc50 [lustre] [ 3256.165430] __fput+0x10d/0x440 [ 3256.167842] ____fput+0x12/0x20 [ 3256.170003] task_work_run+0xc6/0x110 [ 3256.173072] exit_to_usermode_loop+0x1dd/0x1f0 [ 3256.174310] do_syscall_64+0x3d6/0x440 [ 3256.175691] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3256.178269] RIP: 0033:0x7f1a63983cd8 [ 3256.181549] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3256.195559] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3256.200674] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3256.208475] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3256.214026] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3256.219166] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3256.222754] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3272.055287] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3272.058318] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 3272.073043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3272.073120] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3272.073127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3272.073130] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3272.073145] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3272.073149] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3272.073154] RAX: 0000000000003501 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3272.073156] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3272.073159] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3272.073161] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3272.073163] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3272.073166] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3272.073169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3272.073172] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3272.073180] Call Trace: [ 3272.073185] [ 3272.073191] ? show_regs.cold.9+0x22/0x2f [ 3272.073213] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3272.073220] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3272.073224] ? watchdog+0x40/0x40 [ 3272.073228] ? hrtimer_interrupt+0x128/0x2c0 [ 3272.073232] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3272.073240] ? apic_timer_interrupt+0xf/0x20 [ 3272.073245] [ 3272.073248] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3272.073253] do_raw_spin_lock+0x177/0x190 [ 3272.073261] _raw_spin_lock+0x21/0x30 [ 3272.073265] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3272.073546] ? do_raw_spin_unlock+0x75/0x190 [ 3272.073551] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3272.073760] ? do_wait_intr+0xf0/0xf0 [ 3272.073767] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3272.073964] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3272.074178] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3272.074404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3272.074516] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3272.074555] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3272.074586] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3272.074614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3272.074700] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3272.074917] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3272.074945] lmv_intent_open+0x338/0x1150 [lmv] [ 3272.074977] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3272.075059] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3272.075139] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3272.075161] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3272.077459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3272.077542] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3272.078714] ? do_raw_spin_unlock+0x75/0x190 [ 3272.078723] ? _raw_spin_unlock+0x12/0x30 [ 3272.078730] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3272.078758] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3272.078779] ll_atomic_open+0x36a/0xd70 [lustre] [ 3272.079185] lookup_open+0x356/0xb40 [ 3272.079272] do_last+0x313/0xfc0 [ 3272.079277] ? nd_jump_root+0xe5/0x160 [ 3272.079282] ? path_init+0x437/0x520 [ 3272.079286] path_openat+0xf7/0x500 [ 3272.079291] ? do_raw_spin_unlock+0x75/0x190 [ 3272.079295] do_filp_open+0x99/0x140 [ 3272.079301] ? getname_flags+0x6e/0x330 [ 3272.079305] ? __check_object_size+0xff/0x256 [ 3272.079313] ? do_raw_spin_unlock+0x75/0x190 [ 3272.079317] ? _raw_spin_unlock+0x12/0x30 [ 3272.079322] do_sys_openat2+0x2b4/0x410 [ 3272.079327] do_sys_open+0x73/0xa0 [ 3272.079331] __x64_sys_openat+0x24/0x30 [ 3272.079334] do_syscall_64+0xc1/0x440 [ 3272.079341] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3272.079347] RIP: 0033:0x7f1a63983332 [ 3272.079354] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3272.079358] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3272.079364] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3272.079366] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3272.079369] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3272.079371] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3272.079373] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3272.520143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3272.571543] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3272.582866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3272.595443] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3272.597442] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3272.618133] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3272.628553] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3272.638694] RDX: 000000000000512f RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3272.642950] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3272.657783] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3272.669294] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3272.673785] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3272.683370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3272.691795] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3272.699426] Call Trace: [ 3272.700537] [ 3272.704251] ? show_regs.cold.9+0x22/0x2f [ 3272.708261] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3272.719640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3272.721233] ? watchdog+0x40/0x40 [ 3272.723455] ? hrtimer_interrupt+0x128/0x2c0 [ 3272.727480] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3272.731038] ? apic_timer_interrupt+0xf/0x20 [ 3272.733477] [ 3272.734376] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3272.739568] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3272.749489] do_raw_spin_lock+0x177/0x190 [ 3272.751092] _raw_spin_lock+0x21/0x30 [ 3272.762735] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3272.769263] obd_mod_free+0x46/0x140 [ptlrpc] [ 3272.775964] mdc_commit_open+0x91/0xb0 [mdc] [ 3272.779525] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3272.786480] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3272.796590] after_reply+0xe2c/0x1780 [ptlrpc] [ 3272.802595] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3272.807260] ? __next_timer_interrupt+0x160/0x160 [ 3272.808467] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3272.810419] ? do_wait_intr+0xf0/0xf0 [ 3272.811910] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3272.813775] kthread+0x1d1/0x200 [ 3272.814717] ? set_kthread_struct+0x70/0x70 [ 3272.816465] ret_from_fork+0x1f/0x30 [ 3276.772220] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1253s! [ 3276.786714] Showing busy workqueues and worker pools: [ 3276.790811] workqueue events: flags=0x0 [ 3276.796319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3276.803974] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3276.825512] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3276.840136] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3276.858883] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3276.882183] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3276.900649] workqueue events_long: flags=0x0 [ 3276.916248] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3276.920358] pending: lu_objects_flush [obdclass] [ 3276.924823] workqueue events_unbound: flags=0x2 [ 3276.931826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3276.937826] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3276.944659] workqueue events_power_efficient: flags=0x80 [ 3276.947820] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3276.953779] pending: neigh_periodic_work, do_cache_clean [ 3276.958163] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3276.967919] pending: neigh_periodic_work, check_lifetime [ 3276.975393] workqueue mm_percpu_wq: flags=0x8 [ 3276.980856] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3276.989349] pending: vmstat_update [ 3276.996773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3277.012306] pending: vmstat_update [ 3277.018634] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3277.024771] pending: vmstat_update [ 3277.026050] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3277.051846] pending: vmstat_update [ 3277.059964] workqueue writeback: flags=0x4a [ 3277.066986] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3277.072568] pending: wb_workfn, wb_workfn [ 3277.083365] workqueue xprtiod: flags=0xa [ 3277.087420] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3277.097543] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3277.111787] workqueue ipv6_addrconf: flags=0x40008 [ 3277.116886] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3277.119114] pending: addrconf_verify_work [ 3277.128529] workqueue knbd0-recv: flags=0x1a [ 3277.132624] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3277.137134] in-flight: 647:recv_work [ 3277.140374] workqueue ptlrpc_pinger: flags=0x2 [ 3277.146329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3277.149525] pending: ptlrpc_pinger_main [ptlrpc] [ 3277.156472] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3267s workers=2 idle: 652 [ 3280.056057] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3280.063666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3280.116857] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3280.128411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3280.136077] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3280.144144] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3280.160639] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3280.166738] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3280.174747] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3280.181503] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3280.189186] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3280.194428] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3280.203870] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3280.213211] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3280.219539] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3280.227428] Call Trace: [ 3280.228606] [ 3280.229820] ? show_regs.cold.9+0x22/0x2f [ 3280.234128] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3280.240887] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3280.247938] ? watchdog+0x40/0x40 [ 3280.249219] ? hrtimer_interrupt+0x128/0x2c0 [ 3280.254104] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3280.256946] ? apic_timer_interrupt+0xf/0x20 [ 3280.260150] [ 3280.261070] ? native_safe_halt+0x2e/0x50 [ 3280.265098] kvm_wait+0xcd/0xe0 [ 3280.267930] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3280.275720] do_raw_spin_lock+0x177/0x190 [ 3280.279000] _raw_spin_lock+0x21/0x30 [ 3280.283844] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3280.291094] ? do_raw_spin_unlock+0x75/0x190 [ 3280.294123] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3280.299764] ? do_wait_intr+0xf0/0xf0 [ 3280.302940] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3280.309009] mdc_close+0x53b/0xd70 [mdc] [ 3280.314473] lmv_close+0x18d/0x490 [lmv] [ 3280.315975] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3280.326082] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3280.331810] ll_md_close+0x274/0x800 [lustre] [ 3280.337076] ll_file_release+0x128/0xc50 [lustre] [ 3280.340765] __fput+0x10d/0x440 [ 3280.343029] ____fput+0x12/0x20 [ 3280.346700] task_work_run+0xc6/0x110 [ 3280.348109] exit_to_usermode_loop+0x1dd/0x1f0 [ 3280.354875] do_syscall_64+0x3d6/0x440 [ 3280.360393] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3280.370382] RIP: 0033:0x7f1a63983cd8 [ 3280.374688] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3280.397514] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3280.402685] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3280.406930] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3280.414895] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3280.423560] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3280.429495] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3284.060153] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3284.064899] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3284.095446] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3284.115025] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3284.125429] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3284.134732] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3284.156567] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3284.162869] RAX: 000000000000743f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3284.176002] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3284.185325] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3284.189056] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3284.199470] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3284.212827] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3284.219960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3284.231018] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3284.237024] Call Trace: [ 3284.239581] [ 3284.240324] ? show_regs.cold.9+0x22/0x2f [ 3284.247281] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3284.249068] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3284.256382] ? watchdog+0x40/0x40 [ 3284.257665] ? hrtimer_interrupt+0x128/0x2c0 [ 3284.259436] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3284.261393] ? apic_timer_interrupt+0xf/0x20 [ 3284.265788] [ 3284.266824] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3284.271019] do_raw_spin_lock+0x177/0x190 [ 3284.278369] _raw_spin_lock+0x21/0x30 [ 3284.282486] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3284.286213] ? do_raw_spin_unlock+0x75/0x190 [ 3284.292322] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3284.294288] ? do_wait_intr+0xf0/0xf0 [ 3284.298664] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3284.306797] mdc_close+0x53b/0xd70 [mdc] [ 3284.308000] lmv_close+0x18d/0x490 [lmv] [ 3284.309000] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3284.311651] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3284.315879] ll_md_close+0x274/0x800 [lustre] [ 3284.319655] ll_file_release+0x128/0xc50 [lustre] [ 3284.322908] __fput+0x10d/0x440 [ 3284.324274] ____fput+0x12/0x20 [ 3284.325954] task_work_run+0xc6/0x110 [ 3284.327891] exit_to_usermode_loop+0x1dd/0x1f0 [ 3284.329503] do_syscall_64+0x3d6/0x440 [ 3284.332462] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3284.336886] RIP: 0033:0x7f1a63983cd8 [ 3284.340010] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3284.354380] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3284.361563] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3284.366265] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3284.379515] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3284.388522] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3284.396522] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3300.058053] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3300.061361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3300.081809] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3300.086857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3300.090366] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3300.093084] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3300.100415] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3300.103555] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3300.106464] RDX: 0000000000002c46 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3300.109388] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3300.112534] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3300.115502] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3300.118386] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3300.121656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3300.124256] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3300.127123] Call Trace: [ 3300.128035] [ 3300.128731] ? show_regs.cold.9+0x22/0x2f [ 3300.130415] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3300.132309] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3300.134246] ? watchdog+0x40/0x40 [ 3300.135527] ? hrtimer_interrupt+0x128/0x2c0 [ 3300.137216] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3300.139307] ? apic_timer_interrupt+0xf/0x20 [ 3300.141281] [ 3300.142206] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3300.144603] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3300.146611] do_raw_spin_lock+0x177/0x190 [ 3300.148307] _raw_spin_lock+0x21/0x30 [ 3300.149995] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3300.152524] obd_mod_free+0x46/0x140 [ptlrpc] [ 3300.154761] mdc_commit_open+0x91/0xb0 [mdc] [ 3300.157540] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3300.161297] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3300.164393] after_reply+0xe2c/0x1780 [ptlrpc] [ 3300.166681] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3300.169774] ? __next_timer_interrupt+0x160/0x160 [ 3300.171814] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3300.174056] ? do_wait_intr+0xf0/0xf0 [ 3300.175658] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3300.177982] kthread+0x1d1/0x200 [ 3300.179180] ? set_kthread_struct+0x70/0x70 [ 3300.180860] ret_from_fork+0x1f/0x30 [ 3304.059088] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3304.070080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3304.098815] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3304.104279] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3304.112250] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3304.117749] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3304.145642] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3304.159289] RAX: 0000000000007c85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3304.164388] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3304.170289] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3304.173990] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3304.177833] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3304.182275] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3304.186573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3304.192172] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3304.199629] Call Trace: [ 3304.201157] [ 3304.203105] ? show_regs.cold.9+0x22/0x2f [ 3304.205143] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3304.211939] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3304.216137] ? watchdog+0x40/0x40 [ 3304.218078] ? hrtimer_interrupt+0x128/0x2c0 [ 3304.226578] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3304.234295] ? apic_timer_interrupt+0xf/0x20 [ 3304.241428] [ 3304.245791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3304.250782] do_raw_spin_lock+0x177/0x190 [ 3304.254374] _raw_spin_lock+0x21/0x30 [ 3304.261563] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3304.267316] ? do_raw_spin_unlock+0x75/0x190 [ 3304.270831] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3304.280513] ? do_wait_intr+0xf0/0xf0 [ 3304.286724] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3304.292385] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3304.296773] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3304.301570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3304.304790] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3304.308421] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3304.311061] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3304.313543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3304.317237] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3304.320532] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3304.323384] lmv_intent_open+0x338/0x1150 [lmv] [ 3304.326549] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3304.328954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3304.331920] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3304.333224] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3304.335660] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3304.337807] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3304.339724] ? do_raw_spin_unlock+0x75/0x190 [ 3304.343792] ? _raw_spin_unlock+0x12/0x30 [ 3304.345249] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3304.348849] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3304.351777] ll_atomic_open+0x36a/0xd70 [lustre] [ 3304.354476] lookup_open+0x356/0xb40 [ 3304.356536] do_last+0x313/0xfc0 [ 3304.357778] ? nd_jump_root+0xe5/0x160 [ 3304.359158] ? path_init+0x437/0x520 [ 3304.361190] path_openat+0xf7/0x500 [ 3304.363402] ? do_raw_spin_unlock+0x75/0x190 [ 3304.366271] do_filp_open+0x99/0x140 [ 3304.367579] ? getname_flags+0x6e/0x330 [ 3304.369361] ? __check_object_size+0xff/0x256 [ 3304.371560] ? do_raw_spin_unlock+0x75/0x190 [ 3304.373288] ? _raw_spin_unlock+0x12/0x30 [ 3304.375273] do_sys_openat2+0x2b4/0x410 [ 3304.376868] do_sys_open+0x73/0xa0 [ 3304.378631] __x64_sys_openat+0x24/0x30 [ 3304.380226] do_syscall_64+0xc1/0x440 [ 3304.381552] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3304.383300] RIP: 0033:0x7f1a63983332 [ 3304.384582] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3304.391705] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3304.394183] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3304.404396] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3304.410245] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3304.415180] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3304.420422] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3307.491648] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1284s! [ 3307.494770] Showing busy workqueues and worker pools: [ 3307.499486] workqueue events: flags=0x0 [ 3307.501265] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3307.508216] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3307.523519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3307.534392] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3307.550036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3307.554534] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3307.561290] workqueue events_long: flags=0x0 [ 3307.565217] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3307.567830] pending: lu_objects_flush [obdclass] [ 3307.573475] workqueue events_unbound: flags=0x2 [ 3307.577159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3307.579593] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3307.591525] workqueue events_power_efficient: flags=0x80 [ 3307.598053] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3307.602183] pending: neigh_periodic_work, do_cache_clean [ 3307.608086] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3307.610543] pending: neigh_periodic_work, check_lifetime [ 3307.617278] workqueue mm_percpu_wq: flags=0x8 [ 3307.620810] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3307.625005] pending: vmstat_update [ 3307.628271] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3307.630538] pending: vmstat_update [ 3307.633697] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3307.638950] pending: vmstat_update [ 3307.640674] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3307.644597] pending: vmstat_update [ 3307.647546] workqueue writeback: flags=0x4a [ 3307.651473] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3307.654104] pending: wb_workfn, wb_workfn [ 3307.657338] workqueue xprtiod: flags=0xa [ 3307.659360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3307.663513] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3307.667452] workqueue ipv6_addrconf: flags=0x40008 [ 3307.673559] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3307.677700] pending: addrconf_verify_work [ 3307.680200] workqueue knbd0-recv: flags=0x1a [ 3307.682585] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3307.686017] in-flight: 647:recv_work [ 3307.690516] workqueue ptlrpc_pinger: flags=0x2 [ 3307.695229] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3307.699364] pending: ptlrpc_pinger_main [ptlrpc] [ 3307.702402] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3297s workers=2 idle: 652 [ 3308.056017] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3308.060308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3308.088596] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3308.094350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3308.099553] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3308.102828] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3308.113407] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3308.118167] RAX: 0000000000004584 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3308.122990] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3308.127382] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3308.132031] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3308.136421] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3308.140925] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3308.146219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3308.149612] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3308.154172] Call Trace: [ 3308.155272] [ 3308.156237] ? show_regs.cold.9+0x22/0x2f [ 3308.158504] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3308.162432] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3308.166449] ? watchdog+0x40/0x40 [ 3308.169294] ? hrtimer_interrupt+0x128/0x2c0 [ 3308.172455] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3308.175938] ? apic_timer_interrupt+0xf/0x20 [ 3308.178816] [ 3308.180223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3308.183940] do_raw_spin_lock+0x177/0x190 [ 3308.186738] _raw_spin_lock+0x21/0x30 [ 3308.188505] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3308.192494] ? do_raw_spin_unlock+0x75/0x190 [ 3308.195573] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3308.199941] ? do_wait_intr+0xf0/0xf0 [ 3308.202384] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3308.206281] mdc_close+0x53b/0xd70 [mdc] [ 3308.208730] lmv_close+0x18d/0x490 [lmv] [ 3308.210836] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3308.213917] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3308.216355] ll_md_close+0x274/0x800 [lustre] [ 3308.219394] ll_file_release+0x128/0xc50 [lustre] [ 3308.224980] __fput+0x10d/0x440 [ 3308.227066] ____fput+0x12/0x20 [ 3308.230301] task_work_run+0xc6/0x110 [ 3308.232127] exit_to_usermode_loop+0x1dd/0x1f0 [ 3308.235288] do_syscall_64+0x3d6/0x440 [ 3308.238261] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3308.241513] RIP: 0033:0x7f1a63983cd8 [ 3308.244532] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3308.258468] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3308.263623] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3308.267448] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3308.273009] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3308.277889] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3308.282803] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3312.062058] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3312.067594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3312.103762] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3312.111923] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3312.118128] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3312.121151] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3312.132628] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3312.136179] RAX: 0000000000007a8a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3312.144130] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3312.152777] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3312.156225] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3312.162758] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3312.168146] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3312.171209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3312.174735] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3312.177268] Call Trace: [ 3312.178708] [ 3312.179546] ? show_regs.cold.9+0x22/0x2f [ 3312.183123] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3312.186171] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3312.197801] ? watchdog+0x40/0x40 [ 3312.198692] ? hrtimer_interrupt+0x128/0x2c0 [ 3312.199782] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3312.210219] ? apic_timer_interrupt+0xf/0x20 [ 3312.215654] [ 3312.216450] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3312.222655] do_raw_spin_lock+0x177/0x190 [ 3312.224716] _raw_spin_lock+0x21/0x30 [ 3312.226424] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3312.230644] ? do_raw_spin_unlock+0x75/0x190 [ 3312.234640] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3312.238340] ? do_wait_intr+0xf0/0xf0 [ 3312.241717] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3312.245262] mdc_close+0x53b/0xd70 [mdc] [ 3312.247564] lmv_close+0x18d/0x490 [lmv] [ 3312.250318] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3312.256144] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3312.258734] ll_md_close+0x274/0x800 [lustre] [ 3312.261646] ll_file_release+0x128/0xc50 [lustre] [ 3312.270871] __fput+0x10d/0x440 [ 3312.273081] ____fput+0x12/0x20 [ 3312.276266] task_work_run+0xc6/0x110 [ 3312.280122] exit_to_usermode_loop+0x1dd/0x1f0 [ 3312.284965] do_syscall_64+0x3d6/0x440 [ 3312.286681] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3312.289898] RIP: 0033:0x7f1a63983cd8 [ 3312.292780] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3312.311762] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3312.319405] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3312.327743] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3312.334161] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3312.336376] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3312.345155] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3328.058040] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3328.060322] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3328.078364] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3328.082729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3328.086092] RIP: 0010:__pv_queued_spin_lock_slowpath+0x185/0x6d0 [ 3328.088745] Code: 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 48 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 <0f> b6 03 84 c0 75 e6 48 83 05 2c d6 dc 02 01 44 89 f8 f0 66 44 0f [ 3328.095751] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3328.099080] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3328.105187] RDX: 0000000000000417 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3328.117899] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3328.120881] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3328.130174] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3328.138876] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3328.149739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3328.155429] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3328.163264] Call Trace: [ 3328.166289] [ 3328.168244] ? show_regs.cold.9+0x22/0x2f [ 3328.170903] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3328.176084] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3328.179065] ? watchdog+0x40/0x40 [ 3328.180941] ? hrtimer_interrupt+0x128/0x2c0 [ 3328.182474] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3328.195150] ? apic_timer_interrupt+0xf/0x20 [ 3328.199063] [ 3328.199898] ? __pv_queued_spin_lock_slowpath+0x185/0x6d0 [ 3328.205072] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3328.206856] do_raw_spin_lock+0x177/0x190 [ 3328.209324] _raw_spin_lock+0x21/0x30 [ 3328.213261] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3328.219133] obd_mod_free+0x46/0x140 [ptlrpc] [ 3328.221144] mdc_commit_open+0x91/0xb0 [mdc] [ 3328.225554] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3328.233254] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3328.236143] after_reply+0xe2c/0x1780 [ptlrpc] [ 3328.239494] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3328.242227] ? __next_timer_interrupt+0x160/0x160 [ 3328.244373] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3328.246767] ? do_wait_intr+0xf0/0xf0 [ 3328.248191] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3328.253185] kthread+0x1d1/0x200 [ 3328.254241] ? set_kthread_struct+0x70/0x70 [ 3328.255567] ret_from_fork+0x1f/0x30 [ 3332.053028] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3332.056840] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3332.076101] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3332.079948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3332.083359] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3332.085149] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3332.091194] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3332.094052] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3332.096443] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3332.099133] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3332.102836] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3332.106065] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3332.109559] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3332.112791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3332.115257] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3332.118271] Call Trace: [ 3332.119346] [ 3332.120236] ? show_regs.cold.9+0x22/0x2f [ 3332.122087] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3332.124033] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3332.126130] ? watchdog+0x40/0x40 [ 3332.127740] ? hrtimer_interrupt+0x128/0x2c0 [ 3332.129555] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3332.131355] ? apic_timer_interrupt+0xf/0x20 [ 3332.133102] [ 3332.134008] ? native_safe_halt+0x2e/0x50 [ 3332.135681] kvm_wait+0xcd/0xe0 [ 3332.137250] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3332.139288] do_raw_spin_lock+0x177/0x190 [ 3332.141074] _raw_spin_lock+0x21/0x30 [ 3332.142636] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3332.145323] ? do_raw_spin_unlock+0x75/0x190 [ 3332.147218] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3332.149351] ? do_wait_intr+0xf0/0xf0 [ 3332.151101] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3332.153601] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3332.156386] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3332.159204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3332.161188] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3332.163432] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3332.166628] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3332.169132] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3332.172670] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3332.177195] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3332.179695] lmv_intent_open+0x338/0x1150 [lmv] [ 3332.181805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3332.183199] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3332.185201] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3332.186567] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3332.188713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3332.190856] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3332.193001] ? do_raw_spin_unlock+0x75/0x190 [ 3332.196161] ? _raw_spin_unlock+0x12/0x30 [ 3332.199466] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3332.201594] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3332.203231] ll_atomic_open+0x36a/0xd70 [lustre] [ 3332.205470] lookup_open+0x356/0xb40 [ 3332.207427] do_last+0x313/0xfc0 [ 3332.209257] ? nd_jump_root+0xe5/0x160 [ 3332.211035] ? path_init+0x437/0x520 [ 3332.212772] path_openat+0xf7/0x500 [ 3332.214426] ? do_raw_spin_unlock+0x75/0x190 [ 3332.216600] do_filp_open+0x99/0x140 [ 3332.218193] ? getname_flags+0x6e/0x330 [ 3332.219691] ? __check_object_size+0xff/0x256 [ 3332.221332] ? do_raw_spin_unlock+0x75/0x190 [ 3332.224056] ? _raw_spin_unlock+0x12/0x30 [ 3332.225965] do_sys_openat2+0x2b4/0x410 [ 3332.228040] do_sys_open+0x73/0xa0 [ 3332.229549] __x64_sys_openat+0x24/0x30 [ 3332.231325] do_syscall_64+0xc1/0x440 [ 3332.233169] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3332.235396] RIP: 0033:0x7f1a63983332 [ 3332.236801] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3332.243581] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3332.246822] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3332.249583] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3332.252596] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3332.255928] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3332.258922] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3336.057947] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3336.066469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3336.108214] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3336.115352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3336.122496] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3336.136536] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3336.153939] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3336.158899] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3336.165242] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3336.169979] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3336.178548] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3336.181361] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3336.187943] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3336.192080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3336.195959] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3336.199171] Call Trace: [ 3336.200270] [ 3336.201126] ? show_regs.cold.9+0x22/0x2f [ 3336.203210] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3336.207989] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3336.210914] ? watchdog+0x40/0x40 [ 3336.212711] ? hrtimer_interrupt+0x128/0x2c0 [ 3336.214768] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3336.216285] ? apic_timer_interrupt+0xf/0x20 [ 3336.226532] [ 3336.227321] ? native_safe_halt+0x2e/0x50 [ 3336.228745] kvm_wait+0xcd/0xe0 [ 3336.232032] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3336.234986] do_raw_spin_lock+0x177/0x190 [ 3336.236816] _raw_spin_lock+0x21/0x30 [ 3336.246141] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3336.249879] ? do_raw_spin_unlock+0x75/0x190 [ 3336.252150] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3336.255744] ? do_wait_intr+0xf0/0xf0 [ 3336.257967] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3336.260772] mdc_close+0x53b/0xd70 [mdc] [ 3336.262682] lmv_close+0x18d/0x490 [lmv] [ 3336.272315] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3336.276944] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3336.282075] ll_md_close+0x274/0x800 [lustre] [ 3336.283847] ll_file_release+0x128/0xc50 [lustre] [ 3336.293981] __fput+0x10d/0x440 [ 3336.299097] ____fput+0x12/0x20 [ 3336.301194] task_work_run+0xc6/0x110 [ 3336.309727] exit_to_usermode_loop+0x1dd/0x1f0 [ 3336.315103] do_syscall_64+0x3d6/0x440 [ 3336.316638] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3336.322612] RIP: 0033:0x7f1a63983cd8 [ 3336.323961] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3336.360653] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3336.374748] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3336.385321] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3336.393102] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3336.404028] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3336.407821] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3338.214628] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1315s! [ 3338.223660] Showing busy workqueues and worker pools: [ 3338.227730] workqueue events: flags=0x0 [ 3338.234303] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3338.241823] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3338.250180] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3338.258090] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3338.267258] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3338.274851] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3338.284305] workqueue events_long: flags=0x0 [ 3338.287516] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3338.291510] pending: lu_objects_flush [obdclass] [ 3338.294623] workqueue events_unbound: flags=0x2 [ 3338.297274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3338.300874] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3338.307771] workqueue events_power_efficient: flags=0x80 [ 3338.311235] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3338.315374] pending: neigh_periodic_work, do_cache_clean [ 3338.318570] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3338.322371] pending: neigh_periodic_work, check_lifetime [ 3338.325592] workqueue mm_percpu_wq: flags=0x8 [ 3338.328172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3338.332255] pending: vmstat_update [ 3338.334347] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3338.338464] pending: vmstat_update [ 3338.340602] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3338.344614] pending: vmstat_update [ 3338.346785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3338.350925] pending: vmstat_update [ 3338.353436] workqueue writeback: flags=0x4a [ 3338.355774] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3338.359123] pending: wb_workfn, wb_workfn [ 3338.361539] workqueue xprtiod: flags=0xa [ 3338.363643] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3338.367279] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3338.371718] workqueue ipv6_addrconf: flags=0x40008 [ 3338.374240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3338.377731] pending: addrconf_verify_work [ 3338.379732] workqueue knbd0-recv: flags=0x1a [ 3338.381787] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3338.385288] in-flight: 647:recv_work [ 3338.387339] workqueue ptlrpc_pinger: flags=0x2 [ 3338.390033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3338.393164] pending: ptlrpc_pinger_main [ptlrpc] [ 3338.395944] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3328s workers=2 idle: 652 [ 3340.057396] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3340.063870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3340.095980] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3340.103717] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3340.109803] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3340.112513] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3340.119913] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3340.124530] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3340.129948] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3340.134997] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3340.141675] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3340.147330] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3340.153961] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3340.157461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3340.160672] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3340.164915] Call Trace: [ 3340.166467] [ 3340.167387] ? show_regs.cold.9+0x22/0x2f [ 3340.169368] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3340.171385] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3340.184746] ? watchdog+0x40/0x40 [ 3340.186079] ? hrtimer_interrupt+0x128/0x2c0 [ 3340.189698] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3340.193310] ? apic_timer_interrupt+0xf/0x20 [ 3340.195769] [ 3340.196928] ? native_safe_halt+0x2e/0x50 [ 3340.201177] kvm_wait+0xcd/0xe0 [ 3340.202559] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3340.205798] do_raw_spin_lock+0x177/0x190 [ 3340.208102] _raw_spin_lock+0x21/0x30 [ 3340.210744] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3340.214503] ? do_raw_spin_unlock+0x75/0x190 [ 3340.217668] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3340.221460] ? do_wait_intr+0xf0/0xf0 [ 3340.223862] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3340.228732] mdc_close+0x53b/0xd70 [mdc] [ 3340.231753] lmv_close+0x18d/0x490 [lmv] [ 3340.233879] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3340.236848] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3340.239829] ll_md_close+0x274/0x800 [lustre] [ 3340.242880] ll_file_release+0x128/0xc50 [lustre] [ 3340.245884] __fput+0x10d/0x440 [ 3340.247161] ____fput+0x12/0x20 [ 3340.255715] task_work_run+0xc6/0x110 [ 3340.262391] exit_to_usermode_loop+0x1dd/0x1f0 [ 3340.267253] do_syscall_64+0x3d6/0x440 [ 3340.269624] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3340.272186] RIP: 0033:0x7f1a63983cd8 [ 3340.274699] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3340.291457] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3340.299053] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3340.304786] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3340.312634] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3340.319986] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3340.327115] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3342.440103] rcu: INFO: rcu_sched self-detected stall on CPU [ 3342.442054] rcu: 3-...!: (756152 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 3342.446713] (t=1320040 jiffies g=269633 q=680) [ 3342.448358] rcu: rcu_sched kthread starved for 1319686 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 3342.452643] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3342.456660] rcu: RCU grace-period kthread stack dump: [ 3342.458953] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3342.462165] Call Trace: [ 3342.462907] __schedule+0x351/0xcb0 [ 3342.464272] schedule+0xc0/0x180 [ 3342.465160] schedule_timeout+0xb4/0x190 [ 3342.466153] ? do_raw_spin_unlock+0x75/0x190 [ 3342.467305] ? __next_timer_interrupt+0x160/0x160 [ 3342.468650] ? prepare_to_swait_event+0xf3/0x140 [ 3342.469842] rcu_gp_kthread+0x8dd/0x15b0 [ 3342.470833] ? force_qs_rnp+0x2c0/0x2c0 [ 3342.472214] kthread+0x1d1/0x200 [ 3342.472981] ? set_kthread_struct+0x70/0x70 [ 3342.474298] ret_from_fork+0x1f/0x30 [ 3342.475690] rcu: Stack dump where RCU GP kthread last ran: [ 3342.477130] Sending NMI from CPU 3 to CPUs 1: [ 3342.479334] NMI backtrace for cpu 1 [ 3342.479339] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3342.479341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3342.479342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3342.479344] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3342.479346] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 [ 3342.479349] RAX: 00000000000035b5 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3342.479350] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3342.479351] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3342.479353] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3342.479354] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3342.479356] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3342.479358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3342.479359] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3342.479361] Call Trace: [ 3342.479362] [ 3342.479362] ? show_regs.cold.9+0x22/0x2f [ 3342.479364] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3342.479365] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3342.479366] ? nmi_handle+0xbd/0x210 [ 3342.479367] ? default_do_nmi+0x4b/0x1b0 [ 3342.479368] ? do_nmi+0x280/0x380 [ 3342.479369] ? end_repeat_nmi+0x16/0x1f [ 3342.479371] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3342.479372] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3342.479373] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3342.479375] [ 3342.479375] do_raw_spin_lock+0x177/0x190 [ 3342.479376] _raw_spin_lock+0x21/0x30 [ 3342.479377] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3342.479378] ? do_raw_spin_unlock+0x75/0x190 [ 3342.479380] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3342.479381] ? do_wait_intr+0xf0/0xf0 [ 3342.479381] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3342.479383] mdc_close+0x53b/0xd70 [mdc] [ 3342.479384] lmv_close+0x18d/0x490 [lmv] [ 3342.479384] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3342.479386] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3342.479387] ll_md_close+0x274/0x800 [lustre] [ 3342.479388] ll_file_release+0x128/0xc50 [lustre] [ 3342.479389] __fput+0x10d/0x440 [ 3342.479390] ____fput+0x12/0x20 [ 3342.479391] task_work_run+0xc6/0x110 [ 3342.479392] exit_to_usermode_loop+0x1dd/0x1f0 [ 3342.479907] do_syscall_64+0x3d6/0x440 [ 3342.479909] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3342.479910] RIP: 0033:0x7f1a63983cd8 [ 3342.479912] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3342.479914] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3342.479917] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3342.479919] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3342.479920] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3342.479921] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3342.479922] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3342.479931] NMI backtrace for cpu 3 [ 3342.692451] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3342.703438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3342.709802] Call Trace: [ 3342.710990] [ 3342.711736] ? dump_stack+0xbb/0x10e [ 3342.714914] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3342.717911] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3342.720334] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3342.723400] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3342.726499] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3342.728953] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3342.731560] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3342.734243] ? update_process_times+0x6d/0xc0 [ 3342.736267] ? tick_sched_handle.isra.18+0x27/0x90 [ 3342.739189] ? tick_sched_timer+0x53/0xb0 [ 3342.741197] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3342.743724] ? tick_sched_handle.isra.18+0x90/0x90 [ 3342.746945] ? hrtimer_interrupt+0x128/0x2c0 [ 3342.749260] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3342.752807] ? apic_timer_interrupt+0xf/0x20 [ 3342.754779] [ 3342.755483] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3342.758787] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3342.760879] ? do_raw_spin_lock+0x177/0x190 [ 3342.762437] ? _raw_spin_lock+0x21/0x30 [ 3342.764446] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3342.766806] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3342.768916] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3342.771042] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3342.773482] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3342.780921] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3342.783371] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3342.786275] ? __next_timer_interrupt+0x160/0x160 [ 3342.788832] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3342.790796] ? do_wait_intr+0xf0/0xf0 [ 3342.792863] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3342.796921] ? kthread+0x1d1/0x200 [ 3342.801966] ? set_kthread_struct+0x70/0x70 [ 3342.806999] ? ret_from_fork+0x1f/0x30 [ 3364.053552] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3364.060784] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3364.095255] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3364.100642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3364.105145] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3364.107934] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3364.125707] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3364.132237] RAX: 0000000000006248 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3364.138931] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3364.142990] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3364.149929] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3364.155359] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3364.163657] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3364.168698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3364.171679] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3364.175740] Call Trace: [ 3364.177103] [ 3364.178257] ? show_regs.cold.9+0x22/0x2f [ 3364.180330] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3364.182995] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3364.184886] ? watchdog+0x40/0x40 [ 3364.186689] ? hrtimer_interrupt+0x128/0x2c0 [ 3364.188831] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3364.191179] ? apic_timer_interrupt+0xf/0x20 [ 3364.193277] [ 3364.194336] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3364.197889] do_raw_spin_lock+0x177/0x190 [ 3364.203371] _raw_spin_lock+0x21/0x30 [ 3364.205376] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3364.207281] ? do_raw_spin_unlock+0x75/0x190 [ 3364.208727] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3364.211598] ? do_wait_intr+0xf0/0xf0 [ 3364.213245] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3364.215823] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3364.218361] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3364.221230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3364.223692] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3364.226573] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3364.228656] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3364.230630] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3364.236453] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3364.239238] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3364.241602] lmv_intent_open+0x338/0x1150 [lmv] [ 3364.244962] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3364.248795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3364.252317] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3364.254246] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3364.257039] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3364.262135] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3364.263786] ? do_raw_spin_unlock+0x75/0x190 [ 3364.267459] ? _raw_spin_unlock+0x12/0x30 [ 3364.269790] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3364.272907] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3364.275645] ll_atomic_open+0x36a/0xd70 [lustre] [ 3364.277956] lookup_open+0x356/0xb40 [ 3364.279575] do_last+0x313/0xfc0 [ 3364.280894] ? nd_jump_root+0xe5/0x160 [ 3364.283220] ? path_init+0x437/0x520 [ 3364.284958] path_openat+0xf7/0x500 [ 3364.286370] ? do_raw_spin_unlock+0x75/0x190 [ 3364.288918] do_filp_open+0x99/0x140 [ 3364.291063] ? getname_flags+0x6e/0x330 [ 3364.293183] ? __check_object_size+0xff/0x256 [ 3364.295155] ? do_raw_spin_unlock+0x75/0x190 [ 3364.297210] ? _raw_spin_unlock+0x12/0x30 [ 3364.299537] do_sys_openat2+0x2b4/0x410 [ 3364.301297] do_sys_open+0x73/0xa0 [ 3364.303013] __x64_sys_openat+0x24/0x30 [ 3364.305157] do_syscall_64+0xc1/0x440 [ 3364.307224] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3364.309862] RIP: 0033:0x7f1a63983332 [ 3364.311659] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3364.323935] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3364.327254] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3364.330495] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3364.333703] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3364.337109] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3364.340076] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3368.056057] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3368.058717] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3368.058864] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3368.058870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3368.058936] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3368.058941] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3368.058944] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3368.058956] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3368.058960] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3368.058964] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3368.058966] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3368.058968] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3368.058970] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3368.058972] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3368.058976] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3368.058978] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3368.058980] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3368.058991] Call Trace: [ 3368.058996] [ 3368.059001] ? show_regs.cold.9+0x22/0x2f [ 3368.059009] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3368.059015] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3368.059020] ? watchdog+0x40/0x40 [ 3368.059024] ? hrtimer_interrupt+0x128/0x2c0 [ 3368.059027] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3368.059033] ? apic_timer_interrupt+0xf/0x20 [ 3368.059038] [ 3368.059041] ? native_safe_halt+0x2e/0x50 [ 3368.059045] kvm_wait+0xcd/0xe0 [ 3368.059052] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3368.059059] do_raw_spin_lock+0x177/0x190 [ 3368.059064] _raw_spin_lock+0x21/0x30 [ 3368.059069] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3368.059445] obd_mod_free+0x46/0x140 [ptlrpc] [ 3368.059660] mdc_commit_open+0x91/0xb0 [mdc] [ 3368.059707] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3368.059910] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3368.060114] after_reply+0xe2c/0x1780 [ptlrpc] [ 3368.060328] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3368.060538] ? __next_timer_interrupt+0x160/0x160 [ 3368.060546] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3368.060752] ? do_wait_intr+0xf0/0xf0 [ 3368.060762] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3368.060981] kthread+0x1d1/0x200 [ 3368.060988] ? set_kthread_struct+0x70/0x70 [ 3368.060992] ret_from_fork+0x1f/0x30 [ 3368.062070] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3368.062141] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3368.062147] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3368.062150] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3368.062160] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3368.062164] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3368.062169] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3368.062171] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3368.062173] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3368.062175] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3368.062177] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3368.062180] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3368.062183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3368.062186] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3368.062194] Call Trace: [ 3368.062198] [ 3368.062202] ? show_regs.cold.9+0x22/0x2f [ 3368.062213] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3368.062219] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3368.062222] ? watchdog+0x40/0x40 [ 3368.062227] ? hrtimer_interrupt+0x128/0x2c0 [ 3368.062231] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3368.062237] ? apic_timer_interrupt+0xf/0x20 [ 3368.062241] [ 3368.062244] ? native_safe_halt+0x2e/0x50 [ 3368.062249] kvm_wait+0xcd/0xe0 [ 3368.062270] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3368.062280] do_raw_spin_lock+0x177/0x190 [ 3368.062284] _raw_spin_lock+0x21/0x30 [ 3368.062288] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3368.062511] ? do_raw_spin_unlock+0x75/0x190 [ 3368.062517] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3368.062717] ? do_wait_intr+0xf0/0xf0 [ 3368.062724] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3368.063210] mdc_close+0x53b/0xd70 [mdc] [ 3368.063251] lmv_close+0x18d/0x490 [lmv] [ 3368.063353] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3368.063469] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3368.063548] ll_md_close+0x274/0x800 [lustre] [ 3368.063629] ll_file_release+0x128/0xc50 [lustre] [ 3368.063710] __fput+0x10d/0x440 [ 3368.063719] ____fput+0x12/0x20 [ 3368.063724] task_work_run+0xc6/0x110 [ 3368.063732] exit_to_usermode_loop+0x1dd/0x1f0 [ 3368.063739] do_syscall_64+0x3d6/0x440 [ 3368.063744] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3368.063750] RIP: 0033:0x7f1a63983cd8 [ 3368.063757] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3368.063760] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3368.063764] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3368.063767] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3368.063769] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3368.063770] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3368.063772] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3368.360566] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3368.382499] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3368.387329] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3368.391000] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3368.393791] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3368.401723] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3368.405551] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3368.409528] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3368.412467] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3368.416986] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3368.419087] BUG: workqueue lockup - pool [ 3368.421373] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3368.421384] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3368.421387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3368.421390] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3368.421399] Call Trace: [ 3368.421406] [ 3368.421411] ? show_regs.cold.9+0x22/0x2f [ 3368.421424] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3368.421431] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3368.421436] ? watchdog+0x40/0x40 [ 3368.421440] ? hrtimer_interrupt+0x128/0x2c0 [ 3368.421445] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3368.421454] ? apic_timer_interrupt+0xf/0x20 [ 3368.421458] [ 3368.421461] ? native_safe_halt+0x2e/0x50 [ 3368.421464] kvm_wait+0xcd/0xe0 [ 3368.421471] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3368.421480] do_raw_spin_lock+0x177/0x190 [ 3368.421484] _raw_spin_lock+0x21/0x30 [ 3368.421488] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3368.421786] ? do_raw_spin_unlock+0x75/0x190 [ 3368.421792] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3368.421990] ? do_wait_intr+0xf0/0xf0 [ 3368.421998] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3368.422189] mdc_close+0x53b/0xd70 [mdc] [ 3368.422227] lmv_close+0x18d/0x490 [lmv] [ 3368.422258] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3368.423067] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3368.423151] ll_md_close+0x274/0x800 [lustre] [ 3368.423236] ll_file_release+0x128/0xc50 [lustre] [ 3368.423907] __fput+0x10d/0x440 [ 3368.423915] ____fput+0x12/0x20 [ 3368.423920] task_work_run+0xc6/0x110 [ 3368.423926] exit_to_usermode_loop+0x1dd/0x1f0 [ 3368.423932] do_syscall_64+0x3d6/0x440 [ 3368.423936] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3368.423958] RIP: 0033:0x7f1a63983cd8 [ 3368.423964] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3368.423968] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3368.423973] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3368.423975] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3368.423976] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3368.423978] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3368.423980] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3368.532665] cpus=0-3 flags=0x4 nice=0 stuck for 1345s! [ 3368.535246] Showing busy workqueues and worker pools: [ 3368.539745] workqueue events: flags=0x0 [ 3368.543207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3368.546545] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3368.550654] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3368.553738] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3368.557943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3368.561015] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3368.565350] workqueue events_long: flags=0x0 [ 3368.567351] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3368.570725] pending: lu_objects_flush [obdclass] [ 3368.574882] workqueue events_unbound: flags=0x2 [ 3368.578198] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3368.582016] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3368.588398] workqueue events_power_efficient: flags=0x80 [ 3368.592012] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3368.594937] pending: neigh_periodic_work, do_cache_clean [ 3368.596819] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3368.599045] pending: neigh_periodic_work, check_lifetime [ 3368.600802] workqueue mm_percpu_wq: flags=0x8 [ 3368.601972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3368.603940] pending: vmstat_update [ 3368.605174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3368.608126] pending: vmstat_update [ 3368.609991] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3368.614104] pending: vmstat_update [ 3368.615207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3368.618962] pending: vmstat_update [ 3368.621214] workqueue writeback: flags=0x4a [ 3368.624022] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3368.626439] pending: wb_workfn, wb_workfn [ 3368.627808] workqueue xprtiod: flags=0xa [ 3368.630070] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3368.631726] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3368.634932] workqueue ipv6_addrconf: flags=0x40008 [ 3368.636960] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3368.641132] pending: addrconf_verify_work [ 3368.645354] workqueue knbd0-recv: flags=0x1a [ 3368.647305] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3368.649724] in-flight: 647:recv_work [ 3368.651027] workqueue ptlrpc_pinger: flags=0x2 [ 3368.652906] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3368.658788] pending: ptlrpc_pinger_main [ptlrpc] [ 3368.663203] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3358s workers=2 idle: 652 [ 3396.053031] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3396.056019] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3396.058183] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3396.058204] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3396.058292] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3396.058297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3396.058301] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3396.058313] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3396.058317] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3396.058322] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3396.058324] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3396.058327] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3396.058329] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3396.058331] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3396.058335] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3396.058338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3396.058341] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3396.058348] Call Trace: [ 3396.058353] [ 3396.058359] ? show_regs.cold.9+0x22/0x2f [ 3396.058370] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3396.058376] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3396.058380] ? watchdog+0x40/0x40 [ 3396.058385] ? hrtimer_interrupt+0x128/0x2c0 [ 3396.058389] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3396.058395] ? apic_timer_interrupt+0xf/0x20 [ 3396.058400] [ 3396.058403] ? native_safe_halt+0x2e/0x50 [ 3396.058407] kvm_wait+0xcd/0xe0 [ 3396.058414] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3396.058422] do_raw_spin_lock+0x177/0x190 [ 3396.058427] _raw_spin_lock+0x21/0x30 [ 3396.058617] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3396.058623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3396.058693] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3396.058698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3396.058701] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3396.058708] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3396.058432] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3396.058976] obd_mod_free+0x46/0x140 [ptlrpc] [ 3396.059176] mdc_commit_open+0x91/0xb0 [mdc] [ 3396.059257] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3396.060131] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3396.060139] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3396.060142] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3396.060143] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3396.060146] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3396.060148] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3396.060150] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3396.060153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3396.060156] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3396.060165] Call Trace: [ 3396.060170] [ 3396.060173] ? show_regs.cold.9+0x22/0x2f [ 3396.060466] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3396.060472] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3396.060476] ? watchdog+0x40/0x40 [ 3396.060479] ? hrtimer_interrupt+0x128/0x2c0 [ 3396.060483] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3396.060489] ? apic_timer_interrupt+0xf/0x20 [ 3396.060493] [ 3396.060496] ? native_safe_halt+0x2e/0x50 [ 3396.060500] kvm_wait+0xcd/0xe0 [ 3396.060506] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3396.060513] do_raw_spin_lock+0x177/0x190 [ 3396.060517] _raw_spin_lock+0x21/0x30 [ 3396.060521] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3396.060803] ? do_raw_spin_unlock+0x75/0x190 [ 3396.060809] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3396.061745] ? do_wait_intr+0xf0/0xf0 [ 3396.061754] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3396.061974] Modules linked in: lustre(O) [ 3396.061020] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3396.062189] after_reply+0xe2c/0x1780 [ptlrpc] [ 3396.063368] mdc_close+0x53b/0xd70 [mdc] [ 3396.063468] lmv_close+0x18d/0x490 [lmv] [ 3396.063547] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3396.063678] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3396.063762] ll_md_close+0x274/0x800 [lustre] [ 3396.063052] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3396.064639] ll_file_release+0x128/0xc50 [lustre] [ 3396.064796] __fput+0x10d/0x440 [ 3396.064805] ____fput+0x12/0x20 [ 3396.064810] task_work_run+0xc6/0x110 [ 3396.064970] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3396.065050] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3396.065061] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3396.065067] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3396.065080] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3396.065084] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 [ 3396.065303] exit_to_usermode_loop+0x1dd/0x1f0 [ 3396.065314] do_syscall_64+0x3d6/0x440 [ 3396.065319] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3396.065325] RIP: 0033:0x7f1a63983cd8 [ 3396.065332] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3396.065335] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3396.065340] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3396.065342] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3396.065344] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3396.065346] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3396.065348] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3396.065397] ? __next_timer_interrupt+0x160/0x160 [ 3396.065466] ORIG_RAX: ffffffffffffff13 [ 3396.065470] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3396.065472] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3396.065474] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3396.065476] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3396.065479] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3396.065481] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3396.065484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3396.065487] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3396.065495] Call Trace: [ 3396.065499] [ 3396.065502] ? show_regs.cold.9+0x22/0x2f [ 3396.065511] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3396.065517] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3396.065520] ? watchdog+0x40/0x40 [ 3396.065525] ? hrtimer_interrupt+0x128/0x2c0 [ 3396.065408] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3396.065529] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3396.065538] ? apic_timer_interrupt+0xf/0x20 [ 3396.065542] [ 3396.065554] ? native_safe_halt+0x2e/0x50 [ 3396.065559] kvm_wait+0xcd/0xe0 [ 3396.065572] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3396.065579] do_raw_spin_lock+0x177/0x190 [ 3396.065584] _raw_spin_lock+0x21/0x30 [ 3396.065588] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3396.066004] ? do_wait_intr+0xf0/0xf0 [ 3396.066013] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3396.067029] kthread+0x1d1/0x200 [ 3396.067036] ? set_kthread_struct+0x70/0x70 [ 3396.067040] ret_from_fork+0x1f/0x30 [ 3396.077600] osp(O) [ 3396.097450] ? do_raw_spin_unlock+0x75/0x190 [ 3396.097471] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3396.097746] ? do_wait_intr+0xf0/0xf0 [ 3396.097754] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3396.097982] mdc_close+0x53b/0xd70 [mdc] [ 3396.098045] lmv_close+0x18d/0x490 [lmv] [ 3396.098081] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3396.098236] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3396.098322] ll_md_close+0x274/0x800 [lustre] [ 3396.098727] ll_file_release+0x128/0xc50 [lustre] [ 3396.098815] __fput+0x10d/0x440 [ 3396.098824] ____fput+0x12/0x20 [ 3396.098828] task_work_run+0xc6/0x110 [ 3396.098835] exit_to_usermode_loop+0x1dd/0x1f0 [ 3396.098843] do_syscall_64+0x3d6/0x440 [ 3396.098847] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3396.098855] RIP: 0033:0x7f1a63983cd8 [ 3396.098862] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3396.098866] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3396.098872] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3396.098874] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3396.098876] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3396.098878] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3396.098880] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3396.557877] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3396.577808] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3396.583013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3396.587108] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3396.589167] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3396.597600] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3396.601295] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3396.604646] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3396.608304] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3396.611614] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3396.615025] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3396.618181] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3396.621895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3396.625269] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3396.628182] Call Trace: [ 3396.629290] [ 3396.630267] ? show_regs.cold.9+0x22/0x2f [ 3396.632460] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3396.634530] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3396.637744] ? watchdog+0x40/0x40 [ 3396.639654] ? hrtimer_interrupt+0x128/0x2c0 [ 3396.641625] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3396.643787] ? apic_timer_interrupt+0xf/0x20 [ 3396.645712] [ 3396.646672] ? native_safe_halt+0x2e/0x50 [ 3396.647961] kvm_wait+0xcd/0xe0 [ 3396.648925] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3396.650875] do_raw_spin_lock+0x177/0x190 [ 3396.652234] _raw_spin_lock+0x21/0x30 [ 3396.653549] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3396.656381] ? do_raw_spin_unlock+0x75/0x190 [ 3396.658293] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3396.660579] ? do_wait_intr+0xf0/0xf0 [ 3396.661881] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3396.664618] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3396.667146] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3396.669906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3396.672187] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3396.674913] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3396.676756] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3396.678438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3396.680898] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3396.683718] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3396.686195] lmv_intent_open+0x338/0x1150 [lmv] [ 3396.688097] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3396.690343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3396.692610] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3396.694876] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3396.697101] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3396.699792] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3396.702225] ? do_raw_spin_unlock+0x75/0x190 [ 3396.704736] ? _raw_spin_unlock+0x12/0x30 [ 3396.706596] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3396.709951] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3396.712394] ll_atomic_open+0x36a/0xd70 [lustre] [ 3396.715103] lookup_open+0x356/0xb40 [ 3396.716605] do_last+0x313/0xfc0 [ 3396.718217] ? nd_jump_root+0xe5/0x160 [ 3396.720020] ? path_init+0x437/0x520 [ 3396.721900] path_openat+0xf7/0x500 [ 3396.723680] ? do_raw_spin_unlock+0x75/0x190 [ 3396.726131] do_filp_open+0x99/0x140 [ 3396.727793] ? getname_flags+0x6e/0x330 [ 3396.729843] ? __check_object_size+0xff/0x256 [ 3396.731786] ? do_raw_spin_unlock+0x75/0x190 [ 3396.733977] ? _raw_spin_unlock+0x12/0x30 [ 3396.735942] do_sys_openat2+0x2b4/0x410 [ 3396.737874] do_sys_open+0x73/0xa0 [ 3396.739432] __x64_sys_openat+0x24/0x30 [ 3396.741367] do_syscall_64+0xc1/0x440 [ 3396.743118] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3396.745648] RIP: 0033:0x7f1a63983332 [ 3396.747232] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3396.756383] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3396.760015] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3396.763425] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3396.767095] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3396.770452] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3396.773769] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3399.145793] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1376s! [ 3399.153589] Showing busy workqueues and worker pools: [ 3399.158563] workqueue events: flags=0x0 [ 3399.161561] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3399.166615] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3399.173932] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3399.179422] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3399.184830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3399.189645] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3399.197090] workqueue events_long: flags=0x0 [ 3399.200221] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3399.205506] pending: lu_objects_flush [obdclass] [ 3399.208322] workqueue events_unbound: flags=0x2 [ 3399.210191] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3399.213881] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3399.222151] workqueue events_power_efficient: flags=0x80 [ 3399.224578] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3399.234241] pending: neigh_periodic_work, do_cache_clean [ 3399.237860] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3399.245088] pending: neigh_periodic_work, check_lifetime [ 3399.250379] workqueue mm_percpu_wq: flags=0x8 [ 3399.256905] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3399.261611] pending: vmstat_update [ 3399.263603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3399.267851] pending: vmstat_update [ 3399.274153] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3399.282498] pending: vmstat_update [ 3399.287235] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3399.295280] pending: vmstat_update [ 3399.296574] workqueue writeback: flags=0x4a [ 3399.306522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3399.317097] pending: wb_workfn, wb_workfn [ 3399.321491] workqueue xprtiod: flags=0xa [ 3399.324879] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3399.337065] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3399.341487] workqueue ipv6_addrconf: flags=0x40008 [ 3399.352282] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3399.356155] pending: addrconf_verify_work [ 3399.360400] workqueue knbd0-recv: flags=0x1a [ 3399.364369] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3399.368720] in-flight: 647:recv_work [ 3399.374075] workqueue ptlrpc_pinger: flags=0x2 [ 3399.376357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3399.387404] pending: ptlrpc_pinger_main [ptlrpc] [ 3399.396939] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3389s workers=2 idle: 652 [ 3424.053012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3424.057035] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3424.057156] Modules linked in: lustre(O) [ 3424.058134] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3424.058142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3424.058207] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3424.058212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3424.058215] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3424.058228] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3424.058233] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3424.058238] RAX: 00000000000050b5 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3424.058240] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3424.058242] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3424.058245] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3424.058247] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3424.058250] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3424.058252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3424.058254] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3424.058263] Call Trace: [ 3424.058279] [ 3424.058283] ? show_regs.cold.9+0x22/0x2f [ 3424.058292] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3424.058297] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3424.058301] ? watchdog+0x40/0x40 [ 3424.058305] ? hrtimer_interrupt+0x128/0x2c0 [ 3424.058309] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3424.058316] ? apic_timer_interrupt+0xf/0x20 [ 3424.058321] [ 3424.058323] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3424.058330] do_raw_spin_lock+0x177/0x190 [ 3424.058334] _raw_spin_lock+0x21/0x30 [ 3424.058338] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3424.058626] ? do_raw_spin_unlock+0x75/0x190 [ 3424.058631] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3424.058831] ? do_wait_intr+0xf0/0xf0 [ 3424.058839] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3424.058967] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3424.058974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3424.059033] mdc_close+0x53b/0xd70 [mdc] [ 3424.059055] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3424.059061] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3424.059063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3424.059076] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3424.059080] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3424.059086] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3424.059088] RDX: 0000000000006bde RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3424.059073] lmv_close+0x18d/0x490 [lmv] [ 3424.059093] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3424.059096] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3424.059098] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3424.059101] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3424.059107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3424.059110] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3424.059117] Call Trace: [ 3424.059121] [ 3424.059124] ? show_regs.cold.9+0x22/0x2f [ 3424.059132] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3424.059136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3424.059139] ? watchdog+0x40/0x40 [ 3424.059143] ? hrtimer_interrupt+0x128/0x2c0 [ 3424.059147] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3424.059152] ? apic_timer_interrupt+0xf/0x20 [ 3424.059156] [ 3424.059159] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3424.059163] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3424.059106] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3424.059170] do_raw_spin_lock+0x177/0x190 [ 3424.059175] _raw_spin_lock+0x21/0x30 [ 3424.059238] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3424.059368] ll_md_close+0x274/0x800 [lustre] [ 3424.059524] ll_file_release+0x128/0xc50 [lustre] [ 3424.059605] __fput+0x10d/0x440 [ 3424.059613] ____fput+0x12/0x20 [ 3424.059618] task_work_run+0xc6/0x110 [ 3424.059626] exit_to_usermode_loop+0x1dd/0x1f0 [ 3424.059632] do_syscall_64+0x3d6/0x440 [ 3424.059637] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3424.059642] RIP: 0033:0x7f1a63983cd8 [ 3424.059648] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3424.059651] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3424.059655] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3424.059657] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3424.059660] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3424.059662] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3424.059664] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3424.059179] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3424.059866] obd_mod_free+0x46/0x140 [ptlrpc] [ 3424.060060] mdc_commit_open+0x91/0xb0 [mdc] [ 3424.060092] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3424.060987] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3424.061187] after_reply+0xe2c/0x1780 [ptlrpc] [ 3424.061526] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3424.061732] ? __next_timer_interrupt+0x160/0x160 [ 3424.061744] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3424.061944] ? do_wait_intr+0xf0/0xf0 [ 3424.061954] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3424.062150] kthread+0x1d1/0x200 [ 3424.062156] ? set_kthread_struct+0x70/0x70 [ 3424.062160] ret_from_fork+0x1f/0x30 [ 3424.063379] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3424.063450] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3424.063457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3424.063460] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3424.063471] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3424.063476] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3424.063481] RAX: 0000000000005194 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3424.063483] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3424.063485] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3424.063487] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3424.063489] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3424.063492] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3424.063495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3424.063497] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3424.063506] Call Trace: [ 3424.063511] [ 3424.063514] ? show_regs.cold.9+0x22/0x2f [ 3424.063522] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3424.063528] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3424.063532] ? watchdog+0x40/0x40 [ 3424.063536] ? hrtimer_interrupt+0x128/0x2c0 [ 3424.063539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3424.063546] ? apic_timer_interrupt+0xf/0x20 [ 3424.063550] [ 3424.063553] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3424.063558] do_raw_spin_lock+0x177/0x190 [ 3424.063562] _raw_spin_lock+0x21/0x30 [ 3424.063567] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3424.063829] ? do_raw_spin_unlock+0x75/0x190 [ 3424.063835] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3424.067144] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [ 3424.071976] ? do_wait_intr+0xf0/0xf0 [ 3424.097134] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3424.097194] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3424.097201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3424.097203] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3424.097216] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3424.097220] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3424.097225] RAX: 00000000000006ef RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3424.097227] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3424.097229] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3424.097231] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3424.097234] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3424.097237] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3424.097241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3424.097243] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3424.097251] Call Trace: [ 3424.097257] [ 3424.097262] ? show_regs.cold.9+0x22/0x2f [ 3424.097271] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3424.097397] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3424.097405] ? watchdog+0x40/0x40 [ 3424.097410] ? hrtimer_interrupt+0x128/0x2c0 [ 3424.097414] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3424.097422] ? apic_timer_interrupt+0xf/0x20 [ 3424.097434] [ 3424.097437] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3424.097443] do_raw_spin_lock+0x177/0x190 [ 3424.097448] _raw_spin_lock+0x21/0x30 [ 3424.097453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3424.097874] ? do_raw_spin_unlock+0x75/0x190 [ 3424.097880] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3424.098113] ? do_wait_intr+0xf0/0xf0 [ 3424.098121] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3424.098856] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3424.099177] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3424.106323] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3424.106877] mdc_close+0x53b/0xd70 [mdc] [ 3424.106919] lmv_close+0x18d/0x490 [lmv] [ 3424.106959] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3424.107086] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3424.107309] ll_md_close+0x274/0x800 [lustre] [ 3424.107481] ll_file_release+0x128/0xc50 [lustre] [ 3424.107574] __fput+0x10d/0x440 [ 3424.107655] ____fput+0x12/0x20 [ 3424.107662] task_work_run+0xc6/0x110 [ 3424.107670] exit_to_usermode_loop+0x1dd/0x1f0 [ 3424.107676] do_syscall_64+0x3d6/0x440 [ 3424.107681] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3424.107690] RIP: 0033:0x7f1a63983cd8 [ 3424.107696] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3424.107700] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3424.107706] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3424.107708] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3424.107710] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3424.107779] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3424.107782] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3424.736121] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3424.740060] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3424.743992] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3424.746302] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3424.750431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3424.752965] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3424.755833] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3424.759971] lmv_intent_open+0x338/0x1150 [lmv] [ 3424.763063] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3424.766849] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3424.770089] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3424.774204] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3424.777247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3424.780532] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3424.783116] ? do_raw_spin_unlock+0x75/0x190 [ 3424.785229] ? _raw_spin_unlock+0x12/0x30 [ 3424.787215] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3424.790007] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3424.792662] ll_atomic_open+0x36a/0xd70 [lustre] [ 3424.795135] lookup_open+0x356/0xb40 [ 3424.797193] do_last+0x313/0xfc0 [ 3424.798572] ? nd_jump_root+0xe5/0x160 [ 3424.801183] ? path_init+0x437/0x520 [ 3424.803010] path_openat+0xf7/0x500 [ 3424.804939] ? do_raw_spin_unlock+0x75/0x190 [ 3424.807066] do_filp_open+0x99/0x140 [ 3424.808964] ? getname_flags+0x6e/0x330 [ 3424.810529] ? __check_object_size+0xff/0x256 [ 3424.812958] ? do_raw_spin_unlock+0x75/0x190 [ 3424.815255] ? _raw_spin_unlock+0x12/0x30 [ 3424.817493] do_sys_openat2+0x2b4/0x410 [ 3424.820029] do_sys_open+0x73/0xa0 [ 3424.821886] __x64_sys_openat+0x24/0x30 [ 3424.825181] do_syscall_64+0xc1/0x440 [ 3424.826876] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3424.830253] RIP: 0033:0x7f1a63983332 [ 3424.832109] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3424.843154] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3424.850979] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3424.855991] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3424.862103] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3424.864464] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3424.867447] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3429.857043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1406s! [ 3429.861214] Showing busy workqueues and worker pools: [ 3429.868084] workqueue events: flags=0x0 [ 3429.870039] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3429.873855] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3429.883830] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3429.891519] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3429.902306] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3429.905224] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3429.911171] workqueue events_long: flags=0x0 [ 3429.916354] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3429.927301] pending: lu_objects_flush [obdclass] [ 3429.933228] workqueue events_unbound: flags=0x2 [ 3429.936099] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3429.943281] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3429.949208] workqueue events_power_efficient: flags=0x80 [ 3429.952421] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3429.959491] pending: neigh_periodic_work, do_cache_clean [ 3429.962940] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3429.966482] pending: neigh_periodic_work, check_lifetime [ 3429.976963] workqueue mm_percpu_wq: flags=0x8 [ 3429.979826] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3429.983330] pending: vmstat_update [ 3429.989301] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3429.994111] pending: vmstat_update [ 3429.997063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3430.004023] pending: vmstat_update [ 3430.006052] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3430.010968] pending: vmstat_update [ 3430.023475] workqueue writeback: flags=0x4a [ 3430.024835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3430.036626] pending: wb_workfn, wb_workfn [ 3430.037840] workqueue xprtiod: flags=0xa [ 3430.045857] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3430.053419] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3430.059066] workqueue ipv6_addrconf: flags=0x40008 [ 3430.063954] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3430.075900] pending: addrconf_verify_work [ 3430.080897] workqueue knbd0-recv: flags=0x1a [ 3430.084650] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3430.088195] in-flight: 647:recv_work [ 3430.091446] workqueue ptlrpc_pinger: flags=0x2 [ 3430.092843] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3430.098051] pending: ptlrpc_pinger_main [ptlrpc] [ 3430.103096] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3420s workers=2 idle: 652 [ 3452.056024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3452.058132] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3452.058189] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3452.058195] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3452.058267] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3452.058273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3452.058276] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3452.058289] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3452.058294] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3452.058298] RAX: 0000000000004692 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3452.058301] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3452.058303] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3452.058305] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3452.058307] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3452.058311] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3452.058314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3452.058316] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3452.058324] Call Trace: [ 3452.058329] [ 3452.058334] ? show_regs.cold.9+0x22/0x2f [ 3452.058344] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3452.058350] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3452.058355] ? watchdog+0x40/0x40 [ 3452.058359] ? hrtimer_interrupt+0x128/0x2c0 [ 3452.058363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3452.058370] ? apic_timer_interrupt+0xf/0x20 [ 3452.058376] [ 3452.058379] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3452.058385] do_raw_spin_lock+0x177/0x190 [ 3452.058389] _raw_spin_lock+0x21/0x30 [ 3452.058394] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3452.058671] ? do_raw_spin_unlock+0x75/0x190 [ 3452.058676] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3452.058883] ? do_wait_intr+0xf0/0xf0 [ 3452.058890] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3452.061189] mdc_close+0x53b/0xd70 [mdc] [ 3452.061232] lmv_close+0x18d/0x490 [lmv] [ 3452.061261] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3452.061369] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3452.061460] ll_md_close+0x274/0x800 [lustre] [ 3452.061543] ll_file_release+0x128/0xc50 [lustre] [ 3452.061627] __fput+0x10d/0x440 [ 3452.061636] ____fput+0x12/0x20 [ 3452.061640] task_work_run+0xc6/0x110 [ 3452.061648] exit_to_usermode_loop+0x1dd/0x1f0 [ 3452.061655] do_syscall_64+0x3d6/0x440 [ 3452.061660] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3452.061668] RIP: 0033:0x7f1a63983cd8 [ 3452.061675] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3452.061679] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3452.061685] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3452.061687] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3452.061689] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3452.061691] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3452.061693] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3452.067772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3452.067839] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3452.067845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3452.067847] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3452.067857] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3452.067860] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3452.067863] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3452.067865] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3452.067866] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3452.067868] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3452.067869] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3452.067872] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3452.067873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3452.067875] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3452.067930] Call Trace: [ 3452.067935] [ 3452.067939] ? show_regs.cold.9+0x22/0x2f [ 3452.067949] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3452.067955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3452.067958] ? watchdog+0x40/0x40 [ 3452.067962] ? hrtimer_interrupt+0x128/0x2c0 [ 3452.067966] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3452.067973] ? apic_timer_interrupt+0xf/0x20 [ 3452.067977] [ 3452.067981] ? native_safe_halt+0x2e/0x50 [ 3452.067985] kvm_wait+0xcd/0xe0 [ 3452.067992] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3452.067999] do_raw_spin_lock+0x177/0x190 [ 3452.068002] _raw_spin_lock+0x21/0x30 [ 3452.068005] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3452.068417] ? do_raw_spin_unlock+0x75/0x190 [ 3452.068422] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3452.068678] ? do_wait_intr+0xf0/0xf0 [ 3452.068684] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3452.075589] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3452.078135] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3452.078995] mdc_close+0x53b/0xd70 [mdc] [ 3452.079042] lmv_close+0x18d/0x490 [lmv] [ 3452.079072] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3452.079308] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3452.079391] ll_md_close+0x274/0x800 [lustre] [ 3452.122320] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3452.125891] ll_file_release+0x128/0xc50 [lustre] [ 3452.132397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3452.138114] __fput+0x10d/0x440 [ 3452.138141] ____fput+0x12/0x20 [ 3452.138146] task_work_run+0xc6/0x110 [ 3452.138154] exit_to_usermode_loop+0x1dd/0x1f0 [ 3452.138162] do_syscall_64+0x3d6/0x440 [ 3452.151749] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3452.156984] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3452.157006] RIP: 0033:0x7f1a63983cd8 [ 3452.157015] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3452.157019] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3452.157026] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3452.157029] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3452.157032] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3452.157034] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3452.157036] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3452.581343] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3452.585582] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3452.588951] RDX: 0000000000002c0d RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3452.592560] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3452.595788] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3452.599659] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3452.603356] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3452.607316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3452.609906] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3452.612637] Call Trace: [ 3452.613437] [ 3452.614414] ? show_regs.cold.9+0x22/0x2f [ 3452.616544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3452.618819] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3452.620849] ? watchdog+0x40/0x40 [ 3452.622442] ? hrtimer_interrupt+0x128/0x2c0 [ 3452.624540] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3452.627327] ? apic_timer_interrupt+0xf/0x20 [ 3452.629238] [ 3452.630132] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3452.632550] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3452.635763] do_raw_spin_lock+0x177/0x190 [ 3452.637528] _raw_spin_lock+0x21/0x30 [ 3452.639387] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3452.643330] obd_mod_free+0x46/0x140 [ptlrpc] [ 3452.646620] mdc_commit_open+0x91/0xb0 [mdc] [ 3452.649027] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3452.651963] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3452.654993] after_reply+0xe2c/0x1780 [ptlrpc] [ 3452.657856] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3452.660969] ? __next_timer_interrupt+0x160/0x160 [ 3452.663567] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3452.665175] ? do_wait_intr+0xf0/0xf0 [ 3452.666343] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3452.668143] kthread+0x1d1/0x200 [ 3452.669233] ? set_kthread_struct+0x70/0x70 [ 3452.670534] ret_from_fork+0x1f/0x30 [ 3456.053009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3456.059196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3456.087878] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3456.094590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3456.097726] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3456.102062] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3456.121717] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3456.126791] RAX: 0000000000005022 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3456.133074] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3456.140803] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3456.146019] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3456.153868] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3456.157021] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3456.166009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3456.168379] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3456.173073] Call Trace: [ 3456.174486] [ 3456.175291] ? show_regs.cold.9+0x22/0x2f [ 3456.179099] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3456.185053] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3456.191779] ? watchdog+0x40/0x40 [ 3456.198281] ? hrtimer_interrupt+0x128/0x2c0 [ 3456.201576] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3456.207368] ? apic_timer_interrupt+0xf/0x20 [ 3456.210277] [ 3456.211353] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3456.217580] do_raw_spin_lock+0x177/0x190 [ 3456.219936] _raw_spin_lock+0x21/0x30 [ 3456.222267] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3456.226243] ? do_raw_spin_unlock+0x75/0x190 [ 3456.231115] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3456.239929] ? do_wait_intr+0xf0/0xf0 [ 3456.244582] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3456.247454] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3456.249052] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3456.254765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3456.257809] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3456.260469] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3456.262110] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3456.267120] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3456.270038] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3456.275268] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3456.277718] lmv_intent_open+0x338/0x1150 [lmv] [ 3456.281843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3456.285066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3456.293724] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3456.297947] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3456.301925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3456.304577] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3456.306996] ? do_raw_spin_unlock+0x75/0x190 [ 3456.308737] ? _raw_spin_unlock+0x12/0x30 [ 3456.313196] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3456.316865] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3456.318205] ll_atomic_open+0x36a/0xd70 [lustre] [ 3456.320234] lookup_open+0x356/0xb40 [ 3456.325162] do_last+0x313/0xfc0 [ 3456.328151] ? nd_jump_root+0xe5/0x160 [ 3456.331586] ? path_init+0x437/0x520 [ 3456.333037] path_openat+0xf7/0x500 [ 3456.335095] ? do_raw_spin_unlock+0x75/0x190 [ 3456.336783] do_filp_open+0x99/0x140 [ 3456.338190] ? getname_flags+0x6e/0x330 [ 3456.341206] ? __check_object_size+0xff/0x256 [ 3456.343072] ? do_raw_spin_unlock+0x75/0x190 [ 3456.344991] ? _raw_spin_unlock+0x12/0x30 [ 3456.346859] do_sys_openat2+0x2b4/0x410 [ 3456.348676] do_sys_open+0x73/0xa0 [ 3456.350016] __x64_sys_openat+0x24/0x30 [ 3456.353984] do_syscall_64+0xc1/0x440 [ 3456.356387] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3456.358696] RIP: 0033:0x7f1a63983332 [ 3456.360009] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3456.374857] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3456.380689] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3456.387282] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3456.389676] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3456.393187] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3456.395769] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3460.576398] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1437s! [ 3460.586664] Showing busy workqueues and worker pools: [ 3460.589562] workqueue events: flags=0x0 [ 3460.595096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3460.609492] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3460.618014] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3460.625793] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3460.632098] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3460.637718] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3460.644495] workqueue events_long: flags=0x0 [ 3460.646952] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3460.651098] pending: lu_objects_flush [obdclass] [ 3460.654493] workqueue events_unbound: flags=0x2 [ 3460.657195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3460.664932] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3460.671263] workqueue events_power_efficient: flags=0x80 [ 3460.672864] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3460.681986] pending: neigh_periodic_work, do_cache_clean [ 3460.685107] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3460.688817] pending: neigh_periodic_work, check_lifetime [ 3460.691729] workqueue mm_percpu_wq: flags=0x8 [ 3460.693704] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3460.696732] pending: vmstat_update [ 3460.699117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3460.702483] pending: vmstat_update [ 3460.704104] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3460.707471] pending: vmstat_update [ 3460.709040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3460.712306] pending: vmstat_update [ 3460.714110] workqueue writeback: flags=0x4a [ 3460.716045] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3460.719581] pending: wb_workfn, wb_workfn [ 3460.722118] workqueue xprtiod: flags=0xa [ 3460.724092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3460.727321] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3460.731000] workqueue ipv6_addrconf: flags=0x40008 [ 3460.732961] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3460.739178] pending: addrconf_verify_work [ 3460.745075] workqueue knbd0-recv: flags=0x1a [ 3460.750125] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3460.754103] in-flight: 647:recv_work [ 3460.756078] workqueue ptlrpc_pinger: flags=0x2 [ 3460.758126] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3460.761714] pending: ptlrpc_pinger_main [ptlrpc] [ 3460.768037] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3450s workers=2 idle: 652 [ 3480.057009] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 3480.059407] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 3480.060151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3480.060291] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3480.060298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3480.060301] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3480.060314] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3480.060318] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3480.060322] RAX: 0000000000003486 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3480.060324] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3480.060326] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3480.060328] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3480.060330] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3480.060332] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3480.060335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3480.060337] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3480.060346] Call Trace: [ 3480.060350] [ 3480.060355] ? show_regs.cold.9+0x22/0x2f [ 3480.060366] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3480.060373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3480.060377] ? watchdog+0x40/0x40 [ 3480.060382] ? hrtimer_interrupt+0x128/0x2c0 [ 3480.060385] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3480.060395] ? apic_timer_interrupt+0xf/0x20 [ 3480.060399] [ 3480.060402] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3480.060407] do_raw_spin_lock+0x177/0x190 [ 3480.060412] _raw_spin_lock+0x21/0x30 [ 3480.060417] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3480.060683] ? do_raw_spin_unlock+0x75/0x190 [ 3480.060689] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3480.061408] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3480.061414] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3480.061484] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3480.061488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3480.061491] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3480.061500] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3480.061503] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3480.061508] RAX: 0000000000006703 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3480.061511] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3480.061513] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3480.061515] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3480.061517] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3480.061520] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3480.061523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3480.061525] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3480.061532] Call Trace: [ 3480.061535] [ 3480.061538] ? show_regs.cold.9+0x22/0x2f [ 3480.061545] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3480.061550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3480.061554] ? watchdog+0x40/0x40 [ 3480.061559] ? hrtimer_interrupt+0x128/0x2c0 [ 3480.061563] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3480.061571] ? apic_timer_interrupt+0xf/0x20 [ 3480.061575] [ 3480.061578] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3480.061583] do_raw_spin_lock+0x177/0x190 [ 3480.061587] _raw_spin_lock+0x21/0x30 [ 3480.061592] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3480.063817] ? do_raw_spin_unlock+0x75/0x190 [ 3480.063826] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3480.064050] ? do_wait_intr+0xf0/0xf0 [ 3480.064057] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3480.064270] mdc_close+0x53b/0xd70 [mdc] [ 3480.064317] lmv_close+0x18d/0x490 [lmv] [ 3480.064349] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3480.064468] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3480.064554] ll_md_close+0x274/0x800 [lustre] [ 3480.065374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 3480.067446] ll_file_release+0x128/0xc50 [lustre] [ 3480.067540] __fput+0x10d/0x440 [ 3480.067551] ____fput+0x12/0x20 [ 3480.067556] task_work_run+0xc6/0x110 [ 3480.067564] exit_to_usermode_loop+0x1dd/0x1f0 [ 3480.067571] do_syscall_64+0x3d6/0x440 [ 3480.067576] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3480.067586] RIP: 0033:0x7f1a63983cd8 [ 3480.067647] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3480.067652] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3480.067658] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3480.067661] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3480.067663] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3480.067665] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3480.067668] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3480.114768] ? do_wait_intr+0xf0/0xf0 [ 3480.114792] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3480.116250] mdc_close+0x53b/0xd70 [mdc] [ 3480.116292] lmv_close+0x18d/0x490 [lmv] [ 3480.116322] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3480.116430] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3480.125680] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3480.125746] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3480.137410] ll_md_close+0x274/0x800 [lustre] [ 3480.141724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3480.141733] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3480.141749] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3480.157758] ll_file_release+0x128/0xc50 [lustre] [ 3480.162786] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3480.162800] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3480.162803] RDX: 0000000000005bf2 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3480.162805] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3480.162807] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3480.162815] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3480.162819] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3480.162822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3480.162824] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3480.162832] Call Trace: [ 3480.162837] [ 3480.162843] ? show_regs.cold.9+0x22/0x2f [ 3480.162860] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3480.162866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3480.162871] ? watchdog+0x40/0x40 [ 3480.162875] ? hrtimer_interrupt+0x128/0x2c0 [ 3480.162880] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3480.162891] ? apic_timer_interrupt+0xf/0x20 [ 3480.162896] [ 3480.162899] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3480.162906] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3480.162915] do_raw_spin_lock+0x177/0x190 [ 3480.162920] _raw_spin_lock+0x21/0x30 [ 3480.162926] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3480.163376] obd_mod_free+0x46/0x140 [ptlrpc] [ 3480.163659] mdc_commit_open+0x91/0xb0 [mdc] [ 3480.163770] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3480.164280] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3480.164532] after_reply+0xe2c/0x1780 [ptlrpc] [ 3480.164970] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3480.165323] ? __next_timer_interrupt+0x160/0x160 [ 3480.165339] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3480.165557] ? do_wait_intr+0xf0/0xf0 [ 3480.165568] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3480.165845] kthread+0x1d1/0x200 [ 3480.165851] ? set_kthread_struct+0x70/0x70 [ 3480.165854] ret_from_fork+0x1f/0x30 [ 3480.811728] __fput+0x10d/0x440 [ 3480.813013] ____fput+0x12/0x20 [ 3480.814458] task_work_run+0xc6/0x110 [ 3480.816690] exit_to_usermode_loop+0x1dd/0x1f0 [ 3480.819023] do_syscall_64+0x3d6/0x440 [ 3480.820948] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3480.824308] RIP: 0033:0x7f1a63983cd8 [ 3480.825716] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3480.835006] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3480.841264] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3480.844237] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3480.848201] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3480.851455] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3480.854933] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3488.053271] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3488.058032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3488.094007] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3488.100829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3488.107461] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3488.111750] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3488.129765] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3488.133889] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3488.137516] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3488.141068] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3488.148759] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3488.155774] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3488.157782] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3488.166781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3488.171748] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3488.182862] Call Trace: [ 3488.184721] [ 3488.185606] ? show_regs.cold.9+0x22/0x2f [ 3488.186899] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3488.192916] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3488.197765] ? watchdog+0x40/0x40 [ 3488.199872] ? hrtimer_interrupt+0x128/0x2c0 [ 3488.204296] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3488.208683] ? apic_timer_interrupt+0xf/0x20 [ 3488.211784] [ 3488.213468] ? native_safe_halt+0x2e/0x50 [ 3488.215671] kvm_wait+0xcd/0xe0 [ 3488.220177] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3488.227629] do_raw_spin_lock+0x177/0x190 [ 3488.229057] _raw_spin_lock+0x21/0x30 [ 3488.235772] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3488.239984] ? do_raw_spin_unlock+0x75/0x190 [ 3488.244089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3488.248524] ? do_wait_intr+0xf0/0xf0 [ 3488.251939] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3488.255249] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3488.260683] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3488.266001] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3488.269974] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3488.275743] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3488.278361] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3488.282113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3488.284606] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3488.287056] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3488.288994] lmv_intent_open+0x338/0x1150 [lmv] [ 3488.290822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3488.293251] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3488.295415] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3488.296738] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3488.298686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3488.300787] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3488.302706] ? do_raw_spin_unlock+0x75/0x190 [ 3488.304291] ? _raw_spin_unlock+0x12/0x30 [ 3488.305847] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3488.307791] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3488.310801] ll_atomic_open+0x36a/0xd70 [lustre] [ 3488.312859] lookup_open+0x356/0xb40 [ 3488.318044] do_last+0x313/0xfc0 [ 3488.323019] ? nd_jump_root+0xe5/0x160 [ 3488.324883] ? path_init+0x437/0x520 [ 3488.327751] path_openat+0xf7/0x500 [ 3488.330713] ? do_raw_spin_unlock+0x75/0x190 [ 3488.338567] do_filp_open+0x99/0x140 [ 3488.345209] ? getname_flags+0x6e/0x330 [ 3488.354700] ? __check_object_size+0xff/0x256 [ 3488.356323] ? do_raw_spin_unlock+0x75/0x190 [ 3488.362301] ? _raw_spin_unlock+0x12/0x30 [ 3488.363587] do_sys_openat2+0x2b4/0x410 [ 3488.364980] do_sys_open+0x73/0xa0 [ 3488.367654] __x64_sys_openat+0x24/0x30 [ 3488.368739] do_syscall_64+0xc1/0x440 [ 3488.374784] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3488.376821] RIP: 0033:0x7f1a63983332 [ 3488.380841] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3488.406049] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3488.411562] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3488.414892] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3488.416729] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3488.424612] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3488.430342] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3490.785047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1467s! [ 3490.790417] Showing busy workqueues and worker pools: [ 3490.792668] workqueue events: flags=0x0 [ 3490.798913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3490.807255] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3490.820886] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3490.826896] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3490.830663] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3490.834242] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3490.841695] workqueue events_long: flags=0x0 [ 3490.845664] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3490.857482] pending: lu_objects_flush [obdclass] [ 3490.860683] workqueue events_unbound: flags=0x2 [ 3490.864510] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3490.872456] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3490.888534] workqueue events_power_efficient: flags=0x80 [ 3490.894555] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3490.901014] pending: neigh_periodic_work, do_cache_clean [ 3490.904935] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3490.910928] pending: neigh_periodic_work, check_lifetime [ 3490.918284] workqueue mm_percpu_wq: flags=0x8 [ 3490.921709] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3490.928951] pending: vmstat_update [ 3490.930947] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3490.940014] pending: vmstat_update [ 3490.943499] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3490.945755] pending: vmstat_update [ 3490.950266] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3490.955610] pending: vmstat_update [ 3490.958826] workqueue writeback: flags=0x4a [ 3490.960014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3490.967586] pending: wb_workfn, wb_workfn [ 3490.969641] workqueue xprtiod: flags=0xa [ 3490.973721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3490.976299] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3490.981221] workqueue ipv6_addrconf: flags=0x40008 [ 3490.987044] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3490.992378] pending: addrconf_verify_work [ 3490.996191] workqueue knbd0-recv: flags=0x1a [ 3490.997646] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3491.002611] in-flight: 647:recv_work [ 3491.008049] workqueue ptlrpc_pinger: flags=0x2 [ 3491.011839] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3491.020784] pending: ptlrpc_pinger_main [ptlrpc] [ 3491.023771] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3481s workers=2 idle: 652 [ 3508.056023] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 3508.059575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [ 3508.059885] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 3508.062130] nfsv4 [ 3508.062325] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3508.062333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3508.062397] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3508.062402] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3508.062405] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3508.062420] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3508.062424] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3508.062428] RAX: 0000000000005210 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3508.062431] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3508.062433] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3508.062435] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3508.062437] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3508.062440] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3508.062443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3508.062445] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3508.062454] Call Trace: [ 3508.062459] [ 3508.062466] ? show_regs.cold.9+0x22/0x2f [ 3508.062475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3508.062482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3508.062486] ? watchdog+0x40/0x40 [ 3508.062490] ? hrtimer_interrupt+0x128/0x2c0 [ 3508.062494] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3508.062502] ? apic_timer_interrupt+0xf/0x20 [ 3508.062507] [ 3508.062509] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3508.062515] do_raw_spin_lock+0x177/0x190 [ 3508.062520] _raw_spin_lock+0x21/0x30 [ 3508.062524] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3508.062799] ? do_raw_spin_unlock+0x75/0x190 [ 3508.062805] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3508.063017] ? do_wait_intr+0xf0/0xf0 [ 3508.063025] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3508.066155] mdc_close+0x53b/0xd70 [mdc] [ 3508.066204] lmv_close+0x18d/0x490 [lmv] [ 3508.066239] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3508.066359] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3508.066448] ll_md_close+0x274/0x800 [lustre] [ 3508.066858] ll_file_release+0x128/0xc50 [lustre] [ 3508.066979] __fput+0x10d/0x440 [ 3508.066990] ____fput+0x12/0x20 [ 3508.066995] task_work_run+0xc6/0x110 [ 3508.067004] exit_to_usermode_loop+0x1dd/0x1f0 [ 3508.067011] do_syscall_64+0x3d6/0x440 [ 3508.067017] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3508.067025] RIP: 0033:0x7f1a63983cd8 [ 3508.067033] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3508.067038] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3508.067043] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3508.067045] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3508.067047] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3508.067049] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3508.067051] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3508.071130] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3508.071208] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3508.071215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3508.071218] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3508.071233] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3508.071237] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3508.071242] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3508.071244] RDX: 00000000000076d3 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3508.071246] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3508.071248] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3508.071250] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3508.071253] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3508.071255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3508.071257] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3508.071266] Call Trace: [ 3508.071272] [ 3508.071278] ? show_regs.cold.9+0x22/0x2f [ 3508.071288] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3508.071295] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3508.071299] ? watchdog+0x40/0x40 [ 3508.071302] ? hrtimer_interrupt+0x128/0x2c0 [ 3508.071306] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3508.071313] ? apic_timer_interrupt+0xf/0x20 [ 3508.071318] [ 3508.071320] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3508.071325] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3508.071330] do_raw_spin_lock+0x177/0x190 [ 3508.071334] _raw_spin_lock+0x21/0x30 [ 3508.071339] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3508.071627] obd_mod_free+0x46/0x140 [ptlrpc] [ 3508.072041] mdc_commit_open+0x91/0xb0 [mdc] [ 3508.072078] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3508.072297] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3508.072491] after_reply+0xe2c/0x1780 [ptlrpc] [ 3508.072839] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3508.073204] ? __next_timer_interrupt+0x160/0x160 [ 3508.073215] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3508.074047] ? do_wait_intr+0xf0/0xf0 [ 3508.074057] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3508.074280] kthread+0x1d1/0x200 [ 3508.074286] ? set_kthread_struct+0x70/0x70 [ 3508.074290] ret_from_fork+0x1f/0x30 [ 3508.405437] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3508.417457] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3508.421462] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3508.424528] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3508.426473] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3508.433480] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3508.436426] RAX: 0000000000007605 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3508.443977] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3508.448698] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3508.453288] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3508.456392] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3508.459075] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3508.464008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3508.466462] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3508.469164] Call Trace: [ 3508.470049] [ 3508.471153] ? show_regs.cold.9+0x22/0x2f [ 3508.474437] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3508.476646] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3508.481282] ? watchdog+0x40/0x40 [ 3508.487886] ? hrtimer_interrupt+0x128/0x2c0 [ 3508.490488] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3508.493758] ? apic_timer_interrupt+0xf/0x20 [ 3508.495354] [ 3508.498814] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3508.503675] do_raw_spin_lock+0x177/0x190 [ 3508.505303] _raw_spin_lock+0x21/0x30 [ 3508.506629] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3508.514362] ? do_raw_spin_unlock+0x75/0x190 [ 3508.516441] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3508.522095] ? do_wait_intr+0xf0/0xf0 [ 3508.523834] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3508.535227] mdc_close+0x53b/0xd70 [mdc] [ 3508.536618] lmv_close+0x18d/0x490 [lmv] [ 3508.538465] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3508.552457] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3508.562306] ll_md_close+0x274/0x800 [lustre] [ 3508.569476] ll_file_release+0x128/0xc50 [lustre] [ 3508.577272] __fput+0x10d/0x440 [ 3508.579824] ____fput+0x12/0x20 [ 3508.582088] task_work_run+0xc6/0x110 [ 3508.584279] exit_to_usermode_loop+0x1dd/0x1f0 [ 3508.586840] do_syscall_64+0x3d6/0x440 [ 3508.594446] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3508.598063] RIP: 0033:0x7f1a63983cd8 [ 3508.601857] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3508.617091] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3508.624064] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3508.629401] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3508.632420] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3508.637831] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3508.644158] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3516.056888] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3516.062536] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3516.106074] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3516.115180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3516.121742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3516.124076] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3516.131929] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3516.135016] RAX: 000000000000531a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3516.137695] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3516.143446] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3516.150126] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3516.153342] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3516.160575] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3516.166807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3516.172513] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3516.175708] Call Trace: [ 3516.176871] [ 3516.177762] ? show_regs.cold.9+0x22/0x2f [ 3516.179764] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3516.181975] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3516.184057] ? watchdog+0x40/0x40 [ 3516.185483] ? hrtimer_interrupt+0x128/0x2c0 [ 3516.187264] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3516.189668] ? apic_timer_interrupt+0xf/0x20 [ 3516.191645] [ 3516.192519] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3516.194959] do_raw_spin_lock+0x177/0x190 [ 3516.196269] _raw_spin_lock+0x21/0x30 [ 3516.198066] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3516.200650] ? do_raw_spin_unlock+0x75/0x190 [ 3516.202957] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3516.208761] ? do_wait_intr+0xf0/0xf0 [ 3516.213309] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3516.220804] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3516.223124] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3516.229350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3516.235474] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3516.239834] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3516.241161] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3516.242645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3516.244987] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3516.247444] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3516.249569] lmv_intent_open+0x338/0x1150 [lmv] [ 3516.251405] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3516.253664] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3516.255910] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3516.257458] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3516.259748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3516.262868] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3516.264808] ? do_raw_spin_unlock+0x75/0x190 [ 3516.266520] ? _raw_spin_unlock+0x12/0x30 [ 3516.268365] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3516.270755] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3516.273152] ll_atomic_open+0x36a/0xd70 [lustre] [ 3516.275644] lookup_open+0x356/0xb40 [ 3516.276967] do_last+0x313/0xfc0 [ 3516.278204] ? nd_jump_root+0xe5/0x160 [ 3516.279673] ? path_init+0x437/0x520 [ 3516.281114] path_openat+0xf7/0x500 [ 3516.282472] ? do_raw_spin_unlock+0x75/0x190 [ 3516.284107] do_filp_open+0x99/0x140 [ 3516.285655] ? getname_flags+0x6e/0x330 [ 3516.287405] ? __check_object_size+0xff/0x256 [ 3516.289223] ? do_raw_spin_unlock+0x75/0x190 [ 3516.291026] ? _raw_spin_unlock+0x12/0x30 [ 3516.292707] do_sys_openat2+0x2b4/0x410 [ 3516.294428] do_sys_open+0x73/0xa0 [ 3516.295809] __x64_sys_openat+0x24/0x30 [ 3516.297482] do_syscall_64+0xc1/0x440 [ 3516.299086] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3516.301223] RIP: 0033:0x7f1a63983332 [ 3516.302740] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3516.310364] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3516.313470] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3516.316893] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3516.319711] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3516.324360] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3516.329162] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3521.505487] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1498s! [ 3521.513996] Showing busy workqueues and worker pools: [ 3521.516300] workqueue events: flags=0x0 [ 3521.518390] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3521.528505] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3521.534546] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3521.543506] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3521.558129] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3521.566192] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3521.581392] workqueue events_long: flags=0x0 [ 3521.583046] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3521.593168] pending: lu_objects_flush [obdclass] [ 3521.597274] workqueue events_unbound: flags=0x2 [ 3521.601111] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3521.604918] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3521.610165] workqueue events_power_efficient: flags=0x80 [ 3521.612713] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3521.616147] pending: neigh_periodic_work, do_cache_clean [ 3521.618534] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3521.622386] pending: neigh_periodic_work, check_lifetime [ 3521.625307] workqueue mm_percpu_wq: flags=0x8 [ 3521.627230] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3521.630290] pending: vmstat_update [ 3521.632530] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3521.644007] pending: vmstat_update [ 3521.646277] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3521.655389] pending: vmstat_update [ 3521.661900] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3521.667448] pending: vmstat_update [ 3521.669956] workqueue writeback: flags=0x4a [ 3521.674567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3521.684235] pending: wb_workfn, wb_workfn [ 3521.687422] workqueue xprtiod: flags=0xa [ 3521.690158] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3521.694602] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3521.698566] workqueue ipv6_addrconf: flags=0x40008 [ 3521.703338] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3521.709538] pending: addrconf_verify_work [ 3521.715746] workqueue knbd0-recv: flags=0x1a [ 3521.719654] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3521.722451] in-flight: 647:recv_work [ 3521.724371] workqueue ptlrpc_pinger: flags=0x2 [ 3521.726593] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3521.730167] pending: ptlrpc_pinger_main [ptlrpc] [ 3521.735025] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3511s workers=2 idle: 652 [ 3522.443148] rcu: INFO: rcu_sched self-detected stall on CPU [ 3522.445935] rcu: 3-...!: (875279 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 3522.451190] (t=1500043 jiffies g=269633 q=709) [ 3522.453360] rcu: rcu_sched kthread starved for 1499692 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 3522.459144] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3522.463778] rcu: RCU grace-period kthread stack dump: [ 3522.466312] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3522.471132] Call Trace: [ 3522.472439] __schedule+0x351/0xcb0 [ 3522.474161] schedule+0xc0/0x180 [ 3522.475574] schedule_timeout+0xb4/0x190 [ 3522.477599] ? do_raw_spin_unlock+0x75/0x190 [ 3522.479496] ? __next_timer_interrupt+0x160/0x160 [ 3522.481627] ? prepare_to_swait_event+0xf3/0x140 [ 3522.484645] rcu_gp_kthread+0x8dd/0x15b0 [ 3522.486578] ? force_qs_rnp+0x2c0/0x2c0 [ 3522.488686] kthread+0x1d1/0x200 [ 3522.490187] ? set_kthread_struct+0x70/0x70 [ 3522.491704] ret_from_fork+0x1f/0x30 [ 3522.493285] rcu: Stack dump where RCU GP kthread last ran: [ 3522.495884] Sending NMI from CPU 3 to CPUs 0: [ 3522.501280] NMI backtrace for cpu 0 [ 3522.501287] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3522.501288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3522.501290] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3522.501292] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3522.501294] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 [ 3522.501300] RAX: 0000000000004df0 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3522.501301] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3522.501302] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3522.501304] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3522.501305] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3522.501307] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3522.501308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3522.501313] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3522.501314] Call Trace: [ 3522.501315] [ 3522.501316] ? show_regs.cold.9+0x22/0x2f [ 3522.501317] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3522.501318] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3522.501319] ? nmi_handle+0xbd/0x210 [ 3522.501320] ? default_do_nmi+0x4b/0x1b0 [ 3522.501321] ? do_nmi+0x280/0x380 [ 3522.501322] ? end_repeat_nmi+0x16/0x1f [ 3522.501323] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3522.501324] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3522.501329] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3522.501330] [ 3522.501331] do_raw_spin_lock+0x177/0x190 [ 3522.501332] _raw_spin_lock+0x21/0x30 [ 3522.501333] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3522.501334] ? do_raw_spin_unlock+0x75/0x190 [ 3522.501335] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3522.501340] ? do_wait_intr+0xf0/0xf0 [ 3522.501341] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3522.501342] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3522.501343] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3522.501345] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3522.501346] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3522.501347] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3522.501348] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3522.501349] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3522.501350] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3522.501352] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3522.501353] lmv_intent_open+0x338/0x1150 [lmv] [ 3522.501354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3522.501355] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3522.501357] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3522.501358] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3522.501359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3522.501361] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3522.501362] ? do_raw_spin_unlock+0x75/0x190 [ 3522.501363] ? _raw_spin_unlock+0x12/0x30 [ 3522.501364] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3522.501365] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3522.501367] ll_atomic_open+0x36a/0xd70 [lustre] [ 3522.501368] lookup_open+0x356/0xb40 [ 3522.501369] do_last+0x313/0xfc0 [ 3522.501370] ? nd_jump_root+0xe5/0x160 [ 3522.501371] ? path_init+0x437/0x520 [ 3522.501372] path_openat+0xf7/0x500 [ 3522.501373] ? do_raw_spin_unlock+0x75/0x190 [ 3522.501374] do_filp_open+0x99/0x140 [ 3522.501379] ? getname_flags+0x6e/0x330 [ 3522.501380] ? __check_object_size+0xff/0x256 [ 3522.501381] ? do_raw_spin_unlock+0x75/0x190 [ 3522.501382] ? _raw_spin_unlock+0x12/0x30 [ 3522.501383] do_sys_openat2+0x2b4/0x410 [ 3522.501385] do_sys_open+0x73/0xa0 [ 3522.501386] __x64_sys_openat+0x24/0x30 [ 3522.501387] do_syscall_64+0xc1/0x440 [ 3522.501388] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3522.501389] RIP: 0033:0x7f1a63983332 [ 3522.501390] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3522.501393] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3522.501395] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3522.501397] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3522.501398] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3522.501400] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3522.501401] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3522.504561] NMI backtrace for cpu 3 [ 3522.709860] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3522.714636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3522.718454] Call Trace: [ 3522.719452] [ 3522.720346] ? dump_stack+0xbb/0x10e [ 3522.721734] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3522.724137] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3522.726070] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3522.728171] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3522.730420] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3522.732730] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3522.735920] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3522.741792] ? update_process_times+0x6d/0xc0 [ 3522.744530] ? tick_sched_handle.isra.18+0x27/0x90 [ 3522.746969] ? tick_sched_timer+0x53/0xb0 [ 3522.749352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3522.753582] ? tick_sched_handle.isra.18+0x90/0x90 [ 3522.759863] ? hrtimer_interrupt+0x128/0x2c0 [ 3522.762247] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3522.766942] ? apic_timer_interrupt+0xf/0x20 [ 3522.771618] [ 3522.774650] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3522.781674] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3522.788668] ? do_raw_spin_lock+0x177/0x190 [ 3522.792657] ? _raw_spin_lock+0x21/0x30 [ 3522.794510] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3522.799829] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3522.805702] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3522.810345] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3522.815681] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3522.820140] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3522.825688] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3522.829736] ? __next_timer_interrupt+0x160/0x160 [ 3522.833886] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3522.835678] ? do_wait_intr+0xf0/0xf0 [ 3522.841544] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3522.846633] ? kthread+0x1d1/0x200 [ 3522.849567] ? set_kthread_struct+0x70/0x70 [ 3522.854959] ? ret_from_fork+0x1f/0x30 [ 3536.056205] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 3536.058433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3536.059531] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3536.091092] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3536.091106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3536.091109] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3536.091122] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3536.091126] RSP: 0018:ffffc38542073af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3536.091132] RAX: 0000000000003bc1 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3536.091135] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3536.091137] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3536.091140] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3536.091143] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3536.091147] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3536.091149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3536.091152] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3536.091160] Call Trace: [ 3536.091164] [ 3536.091170] ? show_regs.cold.9+0x22/0x2f [ 3536.091179] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3536.091186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3536.091190] ? watchdog+0x40/0x40 [ 3536.091194] ? hrtimer_interrupt+0x128/0x2c0 [ 3536.091199] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3536.091206] ? apic_timer_interrupt+0xf/0x20 [ 3536.091211] [ 3536.091214] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3536.091221] do_raw_spin_lock+0x177/0x190 [ 3536.091226] _raw_spin_lock+0x21/0x30 [ 3536.091230] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3536.092592] ? do_raw_spin_unlock+0x75/0x190 [ 3536.092601] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3536.094773] ? do_wait_intr+0xf0/0xf0 [ 3536.094784] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3536.095007] mdc_close+0x53b/0xd70 [mdc] [ 3536.095045] lmv_close+0x18d/0x490 [lmv] [ 3536.095078] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3536.106924] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3536.107002] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3536.107010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3536.107013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3536.107028] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3536.107032] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3536.107036] RAX: 0000000000003e2e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3536.107039] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3536.107041] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3536.107043] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3536.107046] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3536.107049] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3536.107051] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3536.107053] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3536.107062] Call Trace: [ 3536.107080] [ 3536.107086] ? show_regs.cold.9+0x22/0x2f [ 3536.107095] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3536.107101] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3536.107104] ? watchdog+0x40/0x40 [ 3536.107108] ? hrtimer_interrupt+0x128/0x2c0 [ 3536.107112] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3536.107119] ? apic_timer_interrupt+0xf/0x20 [ 3536.107123] [ 3536.107126] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3536.107131] do_raw_spin_lock+0x177/0x190 [ 3536.107136] _raw_spin_lock+0x21/0x30 [ 3536.107141] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3536.107406] ? do_raw_spin_unlock+0x75/0x190 [ 3536.107412] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3536.108024] ? do_wait_intr+0xf0/0xf0 [ 3536.108033] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3536.108303] mdc_close+0x53b/0xd70 [mdc] [ 3536.108344] lmv_close+0x18d/0x490 [lmv] [ 3536.108375] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3536.108984] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3536.109061] ll_md_close+0x274/0x800 [lustre] [ 3536.109257] ll_file_release+0x128/0xc50 [lustre] [ 3536.109337] __fput+0x10d/0x440 [ 3536.109347] ____fput+0x12/0x20 [ 3536.109351] task_work_run+0xc6/0x110 [ 3536.109358] exit_to_usermode_loop+0x1dd/0x1f0 [ 3536.109365] do_syscall_64+0x3d6/0x440 [ 3536.109368] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3536.109376] RIP: 0033:0x7f1a63983cd8 [ 3536.109383] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3536.109386] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3536.109391] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3536.109393] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3536.109395] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3536.109397] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3536.109399] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3536.476544] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3536.478889] ll_md_close+0x274/0x800 [lustre] [ 3536.481058] ll_file_release+0x128/0xc50 [lustre] [ 3536.484126] __fput+0x10d/0x440 [ 3536.485497] ____fput+0x12/0x20 [ 3536.486943] task_work_run+0xc6/0x110 [ 3536.488518] exit_to_usermode_loop+0x1dd/0x1f0 [ 3536.490589] do_syscall_64+0x3d6/0x440 [ 3536.492625] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3536.495156] RIP: 0033:0x7f1a63983cd8 [ 3536.497175] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3536.508052] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3536.513058] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3536.518157] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3536.528325] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3536.543217] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3536.550509] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3548.053051] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3548.056117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3548.058017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3548.102440] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3548.105426] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3548.105509] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3548.105516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3548.105528] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3548.105542] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3548.105546] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3548.105551] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3548.105553] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3548.105556] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3548.105558] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3548.105560] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3548.105563] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3548.105566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3548.105568] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3548.105578] Call Trace: [ 3548.105582] [ 3548.105587] ? show_regs.cold.9+0x22/0x2f [ 3548.105597] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3548.105604] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3548.105607] ? watchdog+0x40/0x40 [ 3548.105612] ? hrtimer_interrupt+0x128/0x2c0 [ 3548.105616] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3548.105622] ? apic_timer_interrupt+0xf/0x20 [ 3548.105627] [ 3548.105630] ? native_safe_halt+0x2e/0x50 [ 3548.105634] kvm_wait+0xcd/0xe0 [ 3548.105642] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3548.105648] do_raw_spin_lock+0x177/0x190 [ 3548.105652] _raw_spin_lock+0x21/0x30 [ 3548.105657] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3548.122805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3548.122815] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3548.122830] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3548.122834] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3548.122841] RAX: 0000000000001883 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3548.122843] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3548.122846] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3548.122848] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3548.122851] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3548.122854] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3548.122857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3548.122859] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3548.122867] Call Trace: [ 3548.122872] [ 3548.122877] ? show_regs.cold.9+0x22/0x2f [ 3548.122886] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3548.122892] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3548.122895] ? watchdog+0x40/0x40 [ 3548.122899] ? hrtimer_interrupt+0x128/0x2c0 [ 3548.122903] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3548.122910] ? apic_timer_interrupt+0xf/0x20 [ 3548.122914] [ 3548.122916] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3548.122923] do_raw_spin_lock+0x177/0x190 [ 3548.122928] _raw_spin_lock+0x21/0x30 [ 3548.122933] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3548.145178] obd_mod_free+0x46/0x140 [ptlrpc] [ 3548.145455] mdc_commit_open+0x91/0xb0 [mdc] [ 3548.145493] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3548.145702] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3548.145901] after_reply+0xe2c/0x1780 [ptlrpc] [ 3548.146099] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3548.146295] ? __next_timer_interrupt+0x160/0x160 [ 3548.146305] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3548.165131] ? do_raw_spin_unlock+0x75/0x190 [ 3548.165153] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3548.165434] ? do_wait_intr+0xf0/0xf0 [ 3548.165443] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3548.170898] ? do_wait_intr+0xf0/0xf0 [ 3548.170922] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3548.171436] kthread+0x1d1/0x200 [ 3548.171442] ? set_kthread_struct+0x70/0x70 [ 3548.171446] ret_from_fork+0x1f/0x30 [ 3548.463926] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3548.467799] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3548.474086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3548.482008] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3548.486266] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3548.491756] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3548.494369] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3548.498201] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3548.505793] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3548.508973] lmv_intent_open+0x338/0x1150 [lmv] [ 3548.510986] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3548.519994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3548.526928] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3548.532346] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3548.536236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3548.540759] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3548.544157] ? do_raw_spin_unlock+0x75/0x190 [ 3548.548686] ? _raw_spin_unlock+0x12/0x30 [ 3548.551041] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3548.552830] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3548.556859] ll_atomic_open+0x36a/0xd70 [lustre] [ 3548.559106] lookup_open+0x356/0xb40 [ 3548.561726] do_last+0x313/0xfc0 [ 3548.564491] ? nd_jump_root+0xe5/0x160 [ 3548.566612] ? path_init+0x437/0x520 [ 3548.568925] path_openat+0xf7/0x500 [ 3548.570819] ? do_raw_spin_unlock+0x75/0x190 [ 3548.573150] do_filp_open+0x99/0x140 [ 3548.574390] ? getname_flags+0x6e/0x330 [ 3548.579855] ? __check_object_size+0xff/0x256 [ 3548.581136] ? do_raw_spin_unlock+0x75/0x190 [ 3548.582175] ? _raw_spin_unlock+0x12/0x30 [ 3548.588370] do_sys_openat2+0x2b4/0x410 [ 3548.589544] do_sys_open+0x73/0xa0 [ 3548.595660] __x64_sys_openat+0x24/0x30 [ 3548.598778] do_syscall_64+0xc1/0x440 [ 3548.603745] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3548.605954] RIP: 0033:0x7f1a63983332 [ 3548.606845] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3548.634239] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3548.641120] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3548.649749] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3548.653485] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3548.656940] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3548.660933] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3552.226237] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1529s! [ 3552.236670] Showing busy workqueues and worker pools: [ 3552.248316] workqueue events: flags=0x0 [ 3552.250880] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3552.257166] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3552.262535] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3552.265581] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3552.270495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3552.274072] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3552.279877] workqueue events_long: flags=0x0 [ 3552.282796] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3552.285820] pending: lu_objects_flush [obdclass] [ 3552.288916] workqueue events_unbound: flags=0x2 [ 3552.291164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3552.298369] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3552.308208] workqueue events_power_efficient: flags=0x80 [ 3552.311275] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3552.314218] pending: neigh_periodic_work, do_cache_clean [ 3552.317809] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3552.322038] pending: neigh_periodic_work, check_lifetime [ 3552.326844] workqueue mm_percpu_wq: flags=0x8 [ 3552.331601] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3552.336375] pending: vmstat_update [ 3552.338442] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3552.343137] pending: vmstat_update [ 3552.345152] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3552.349347] pending: vmstat_update [ 3552.351062] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3552.354695] pending: vmstat_update [ 3552.356311] workqueue writeback: flags=0x4a [ 3552.358403] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3552.362789] pending: wb_workfn, wb_workfn [ 3552.367146] workqueue xprtiod: flags=0xa [ 3552.368680] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3552.373707] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3552.377754] workqueue ipv6_addrconf: flags=0x40008 [ 3552.380557] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3552.384274] pending: addrconf_verify_work [ 3552.386298] workqueue knbd0-recv: flags=0x1a [ 3552.388219] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3552.395342] in-flight: 647:recv_work [ 3552.401386] workqueue ptlrpc_pinger: flags=0x2 [ 3552.404640] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3552.407892] pending: ptlrpc_pinger_main [ptlrpc] [ 3552.412128] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3542s workers=2 idle: 652 [ 3564.056102] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 3564.063151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3564.065817] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3564.084907] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3564.084920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3564.084923] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3564.084937] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3564.084942] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3564.084947] RAX: 0000000000007a8e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3564.084950] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3564.084952] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3564.084955] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3564.084957] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3564.084960] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3564.084964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3564.084966] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3564.084973] Call Trace: [ 3564.084978] [ 3564.084984] ? show_regs.cold.9+0x22/0x2f [ 3564.085056] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3564.085066] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3564.085071] ? watchdog+0x40/0x40 [ 3564.085075] ? hrtimer_interrupt+0x128/0x2c0 [ 3564.085079] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3564.085087] ? apic_timer_interrupt+0xf/0x20 [ 3564.085092] [ 3564.085095] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3564.085102] do_raw_spin_lock+0x177/0x190 [ 3564.085107] _raw_spin_lock+0x21/0x30 [ 3564.085112] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3564.085535] ? do_raw_spin_unlock+0x75/0x190 [ 3564.085541] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3564.085810] ? do_wait_intr+0xf0/0xf0 [ 3564.085818] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3564.086084] mdc_close+0x53b/0xd70 [mdc] [ 3564.086123] lmv_close+0x18d/0x490 [lmv] [ 3564.086151] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3564.086348] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3564.086505] ll_md_close+0x274/0x800 [lustre] [ 3564.086656] ll_file_release+0x128/0xc50 [lustre] [ 3564.086738] __fput+0x10d/0x440 [ 3564.086746] ____fput+0x12/0x20 [ 3564.086750] task_work_run+0xc6/0x110 [ 3564.086759] exit_to_usermode_loop+0x1dd/0x1f0 [ 3564.086765] do_syscall_64+0x3d6/0x440 [ 3564.086768] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3564.086775] RIP: 0033:0x7f1a63983cd8 [ 3564.086782] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3564.086785] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3564.086790] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3564.086792] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3564.086793] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3564.086795] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3564.086797] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3564.361812] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3564.412934] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3564.419893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3564.425873] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3564.428126] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3564.442569] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3564.445956] RAX: 0000000000005784 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3564.449474] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3564.452580] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3564.456707] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3564.460564] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3564.464286] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3564.471158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3564.472871] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3564.480680] Call Trace: [ 3564.482135] [ 3564.482871] ? show_regs.cold.9+0x22/0x2f [ 3564.484293] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3564.488512] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3564.492713] ? watchdog+0x40/0x40 [ 3564.494411] ? hrtimer_interrupt+0x128/0x2c0 [ 3564.496942] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3564.505054] ? apic_timer_interrupt+0xf/0x20 [ 3564.511394] [ 3564.512012] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3564.516064] do_raw_spin_lock+0x177/0x190 [ 3564.517821] _raw_spin_lock+0x21/0x30 [ 3564.520038] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3564.522268] ? do_raw_spin_unlock+0x75/0x190 [ 3564.524268] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3564.529532] ? do_wait_intr+0xf0/0xf0 [ 3564.532218] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3564.535763] mdc_close+0x53b/0xd70 [mdc] [ 3564.537539] lmv_close+0x18d/0x490 [lmv] [ 3564.539497] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3564.544120] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3564.547469] ll_md_close+0x274/0x800 [lustre] [ 3564.554826] ll_file_release+0x128/0xc50 [lustre] [ 3564.557493] __fput+0x10d/0x440 [ 3564.559010] ____fput+0x12/0x20 [ 3564.560259] task_work_run+0xc6/0x110 [ 3564.562140] exit_to_usermode_loop+0x1dd/0x1f0 [ 3564.564547] do_syscall_64+0x3d6/0x440 [ 3564.565806] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3564.572016] RIP: 0033:0x7f1a63983cd8 [ 3564.575275] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3564.587327] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3564.590487] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3564.593794] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3564.596664] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3564.599271] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3564.601856] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3576.058054] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3576.061100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3576.079833] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3576.084382] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3576.087603] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3576.091575] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3576.100508] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3576.104092] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3576.106729] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3576.109932] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3576.113099] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3576.118603] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3576.121273] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3576.125117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3576.127179] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3576.130533] Call Trace: [ 3576.131649] [ 3576.132561] ? show_regs.cold.9+0x22/0x2f [ 3576.134223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3576.135838] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3576.137594] ? watchdog+0x40/0x40 [ 3576.138728] ? hrtimer_interrupt+0x128/0x2c0 [ 3576.140409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3576.142379] ? apic_timer_interrupt+0xf/0x20 [ 3576.143937] [ 3576.145145] ? native_safe_halt+0x2e/0x50 [ 3576.146607] kvm_wait+0xcd/0xe0 [ 3576.147767] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3576.149755] do_raw_spin_lock+0x177/0x190 [ 3576.151295] _raw_spin_lock+0x21/0x30 [ 3576.153102] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3576.155395] obd_mod_free+0x46/0x140 [ptlrpc] [ 3576.157440] mdc_commit_open+0x91/0xb0 [mdc] [ 3576.159366] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3576.162110] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3576.164708] after_reply+0xe2c/0x1780 [ptlrpc] [ 3576.166871] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3576.169121] ? __next_timer_interrupt+0x160/0x160 [ 3576.171083] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3576.173235] ? do_wait_intr+0xf0/0xf0 [ 3576.174717] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3576.176932] kthread+0x1d1/0x200 [ 3576.178158] ? set_kthread_struct+0x70/0x70 [ 3576.179817] ret_from_fork+0x1f/0x30 [ 3580.053610] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3580.061759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.087941] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.092601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.096176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.098805] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3580.106749] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3580.109923] RAX: 0000000000004460 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3580.112790] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3580.116009] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3580.118951] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3580.121959] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3580.125004] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3580.128785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.131297] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3580.134385] Call Trace: [ 3580.135367] [ 3580.135916] ? show_regs.cold.9+0x22/0x2f [ 3580.137198] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.138602] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.140728] ? watchdog+0x40/0x40 [ 3580.142065] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.143841] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.145819] ? apic_timer_interrupt+0xf/0x20 [ 3580.147711] [ 3580.148649] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.150961] do_raw_spin_lock+0x177/0x190 [ 3580.152772] _raw_spin_lock+0x21/0x30 [ 3580.154061] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3580.156459] ? do_raw_spin_unlock+0x75/0x190 [ 3580.158173] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3580.160583] ? do_wait_intr+0xf0/0xf0 [ 3580.162147] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3580.164561] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3580.167016] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3580.169663] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3580.172068] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3580.174488] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3580.176505] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3580.178497] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3580.180752] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3580.183387] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3580.184877] lmv_intent_open+0x338/0x1150 [lmv] [ 3580.186106] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3580.188496] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3580.190775] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3580.192801] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3580.194709] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3580.197069] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3580.199341] ? do_raw_spin_unlock+0x75/0x190 [ 3580.201467] ? _raw_spin_unlock+0x12/0x30 [ 3580.203435] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3580.205972] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3580.208469] ll_atomic_open+0x36a/0xd70 [lustre] [ 3580.210806] lookup_open+0x356/0xb40 [ 3580.212405] do_last+0x313/0xfc0 [ 3580.213783] ? nd_jump_root+0xe5/0x160 [ 3580.215513] ? path_init+0x437/0x520 [ 3580.216897] path_openat+0xf7/0x500 [ 3580.217909] ? do_raw_spin_unlock+0x75/0x190 [ 3580.219722] do_filp_open+0x99/0x140 [ 3580.221175] ? getname_flags+0x6e/0x330 [ 3580.222745] ? __check_object_size+0xff/0x256 [ 3580.224558] ? do_raw_spin_unlock+0x75/0x190 [ 3580.226088] ? _raw_spin_unlock+0x12/0x30 [ 3580.227789] do_sys_openat2+0x2b4/0x410 [ 3580.229560] do_sys_open+0x73/0xa0 [ 3580.230984] __x64_sys_openat+0x24/0x30 [ 3580.232700] do_syscall_64+0xc1/0x440 [ 3580.233790] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3580.235065] RIP: 0033:0x7f1a63983332 [ 3580.235970] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3580.244218] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3580.247884] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3580.251128] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3580.253946] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3580.256917] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3580.259945] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3582.432200] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1559s! [ 3582.436499] Showing busy workqueues and worker pools: [ 3582.438609] workqueue events: flags=0x0 [ 3582.440631] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3582.444129] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3582.448604] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3582.452292] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3582.456594] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3582.459157] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3582.462878] workqueue events_long: flags=0x0 [ 3582.466107] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3582.468679] pending: lu_objects_flush [obdclass] [ 3582.470614] workqueue events_unbound: flags=0x2 [ 3582.472336] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3582.475380] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3582.480860] workqueue events_power_efficient: flags=0x80 [ 3582.483296] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3582.485867] pending: neigh_periodic_work, do_cache_clean [ 3582.488210] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3582.490867] pending: neigh_periodic_work, check_lifetime [ 3582.493522] workqueue mm_percpu_wq: flags=0x8 [ 3582.495707] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3582.499386] pending: vmstat_update [ 3582.500924] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3582.505068] pending: vmstat_update [ 3582.507653] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3582.511684] pending: vmstat_update [ 3582.514422] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3582.517560] pending: vmstat_update [ 3582.518903] workqueue writeback: flags=0x4a [ 3582.521923] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3582.525599] pending: wb_workfn, wb_workfn [ 3582.527902] workqueue xprtiod: flags=0xa [ 3582.530175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3582.533793] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3582.538213] workqueue ipv6_addrconf: flags=0x40008 [ 3582.541023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3582.544548] pending: addrconf_verify_work [ 3582.546297] workqueue knbd0-recv: flags=0x1a [ 3582.547811] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3582.550565] in-flight: 647:recv_work [ 3582.552114] workqueue ptlrpc_pinger: flags=0x2 [ 3582.553862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3582.556921] pending: ptlrpc_pinger_main [ptlrpc] [ 3582.560068] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3572s workers=2 idle: 652 [ 3592.056255] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3592.059442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3592.060178] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3592.081742] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3592.081755] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3592.081759] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3592.081771] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3592.081775] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3592.081780] RAX: 00000000000058ff RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3592.081782] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3592.081784] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3592.081786] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3592.081788] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3592.081791] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3592.081793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3592.081795] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3592.081803] Call Trace: [ 3592.081807] [ 3592.081813] ? show_regs.cold.9+0x22/0x2f [ 3592.081821] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3592.081828] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3592.081832] ? watchdog+0x40/0x40 [ 3592.081836] ? hrtimer_interrupt+0x128/0x2c0 [ 3592.081840] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3592.081847] ? apic_timer_interrupt+0xf/0x20 [ 3592.081850] [ 3592.081853] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3592.081859] do_raw_spin_lock+0x177/0x190 [ 3592.081863] _raw_spin_lock+0x21/0x30 [ 3592.081867] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3592.088596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [ 3592.093818] ? do_raw_spin_unlock+0x75/0x190 [ 3592.093837] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3592.094130] ? do_wait_intr+0xf0/0xf0 [ 3592.094138] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3592.094431] mdc_close+0x53b/0xd70 [mdc] [ 3592.094484] lmv_close+0x18d/0x490 [lmv] [ 3592.094512] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3592.094630] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3592.094710] ll_md_close+0x274/0x800 [lustre] [ 3592.094866] ll_file_release+0x128/0xc50 [lustre] [ 3592.094946] __fput+0x10d/0x440 [ 3592.094956] ____fput+0x12/0x20 [ 3592.094960] task_work_run+0xc6/0x110 [ 3592.094967] exit_to_usermode_loop+0x1dd/0x1f0 [ 3592.094973] do_syscall_64+0x3d6/0x440 [ 3592.094977] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3592.094985] RIP: 0033:0x7f1a63983cd8 [ 3592.094992] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3592.094996] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3592.095001] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3592.095003] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3592.095005] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3592.095007] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3592.095009] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3592.246648] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3592.265485] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3592.276334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3592.281192] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3592.283876] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3592.292732] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3592.296125] RAX: 0000000000005d86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3592.299312] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3592.303405] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3592.306324] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3592.309455] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3592.314346] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3592.318644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3592.322617] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3592.326606] Call Trace: [ 3592.327759] [ 3592.328825] ? show_regs.cold.9+0x22/0x2f [ 3592.330618] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3592.333187] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3592.335372] ? watchdog+0x40/0x40 [ 3592.337625] ? hrtimer_interrupt+0x128/0x2c0 [ 3592.339744] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3592.343355] ? apic_timer_interrupt+0xf/0x20 [ 3592.345971] [ 3592.346925] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3592.350401] do_raw_spin_lock+0x177/0x190 [ 3592.352502] _raw_spin_lock+0x21/0x30 [ 3592.354380] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3592.357462] ? do_raw_spin_unlock+0x75/0x190 [ 3592.359816] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3592.364613] ? do_wait_intr+0xf0/0xf0 [ 3592.367253] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3592.370556] mdc_close+0x53b/0xd70 [mdc] [ 3592.372485] lmv_close+0x18d/0x490 [lmv] [ 3592.374278] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3592.377248] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3592.380289] ll_md_close+0x274/0x800 [lustre] [ 3592.383002] ll_file_release+0x128/0xc50 [lustre] [ 3592.386412] __fput+0x10d/0x440 [ 3592.387864] ____fput+0x12/0x20 [ 3592.389916] task_work_run+0xc6/0x110 [ 3592.391669] exit_to_usermode_loop+0x1dd/0x1f0 [ 3592.393921] do_syscall_64+0x3d6/0x440 [ 3592.396057] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3592.398455] RIP: 0033:0x7f1a63983cd8 [ 3592.400346] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3592.408925] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3592.412761] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3592.415814] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3592.420520] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3592.424877] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3592.428491] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3604.061440] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3604.066238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3604.124439] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3604.137544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3604.149225] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3604.157148] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3604.170558] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3604.175899] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3604.181587] RDX: 00000000000020b0 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3604.185528] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3604.188362] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3604.197547] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3604.203034] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3604.210511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3604.212233] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3604.214416] Call Trace: [ 3604.215385] [ 3604.216336] ? show_regs.cold.9+0x22/0x2f [ 3604.217880] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3604.219937] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3604.221578] ? watchdog+0x40/0x40 [ 3604.222842] ? hrtimer_interrupt+0x128/0x2c0 [ 3604.224273] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3604.226452] ? apic_timer_interrupt+0xf/0x20 [ 3604.228152] [ 3604.229372] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3604.232477] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3604.234512] do_raw_spin_lock+0x177/0x190 [ 3604.236332] _raw_spin_lock+0x21/0x30 [ 3604.237880] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3604.240560] obd_mod_free+0x46/0x140 [ptlrpc] [ 3604.248329] mdc_commit_open+0x91/0xb0 [mdc] [ 3604.250391] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3604.259524] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3604.263561] after_reply+0xe2c/0x1780 [ptlrpc] [ 3604.267589] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3604.269748] ? __next_timer_interrupt+0x160/0x160 [ 3604.271956] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3604.275071] ? do_wait_intr+0xf0/0xf0 [ 3604.276851] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3604.278833] kthread+0x1d1/0x200 [ 3604.282406] ? set_kthread_struct+0x70/0x70 [ 3604.284236] ret_from_fork+0x1f/0x30 [ 3608.053032] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3608.056784] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3608.084101] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3608.090870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3608.095242] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3608.099786] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3608.109279] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3608.113943] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3608.117895] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3608.122044] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3608.125952] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3608.129838] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3608.136249] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3608.142129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3608.145736] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3608.149880] Call Trace: [ 3608.150916] [ 3608.152060] ? show_regs.cold.9+0x22/0x2f [ 3608.153680] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3608.155869] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3608.157895] ? watchdog+0x40/0x40 [ 3608.161003] ? hrtimer_interrupt+0x128/0x2c0 [ 3608.165182] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3608.168061] ? apic_timer_interrupt+0xf/0x20 [ 3608.173561] [ 3608.174234] ? native_safe_halt+0x2e/0x50 [ 3608.175571] kvm_wait+0xcd/0xe0 [ 3608.179547] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3608.181925] do_raw_spin_lock+0x177/0x190 [ 3608.183747] _raw_spin_lock+0x21/0x30 [ 3608.184888] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3608.192039] ? do_raw_spin_unlock+0x75/0x190 [ 3608.194230] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3608.197659] ? do_wait_intr+0xf0/0xf0 [ 3608.198851] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3608.205718] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3608.208863] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3608.211815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3608.214227] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3608.216251] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3608.217844] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3608.219342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3608.221451] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3608.223910] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3608.225906] lmv_intent_open+0x338/0x1150 [lmv] [ 3608.227613] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3608.229599] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3608.231771] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3608.233338] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3608.235028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3608.236734] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3608.238271] ? do_raw_spin_unlock+0x75/0x190 [ 3608.239702] ? _raw_spin_unlock+0x12/0x30 [ 3608.240991] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3608.245064] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3608.246845] ll_atomic_open+0x36a/0xd70 [lustre] [ 3608.249022] lookup_open+0x356/0xb40 [ 3608.252179] do_last+0x313/0xfc0 [ 3608.255787] ? nd_jump_root+0xe5/0x160 [ 3608.257789] ? path_init+0x437/0x520 [ 3608.259124] path_openat+0xf7/0x500 [ 3608.260947] ? do_raw_spin_unlock+0x75/0x190 [ 3608.264700] do_filp_open+0x99/0x140 [ 3608.268166] ? getname_flags+0x6e/0x330 [ 3608.270540] ? __check_object_size+0xff/0x256 [ 3608.271874] ? do_raw_spin_unlock+0x75/0x190 [ 3608.273827] ? _raw_spin_unlock+0x12/0x30 [ 3608.275908] do_sys_openat2+0x2b4/0x410 [ 3608.278154] do_sys_open+0x73/0xa0 [ 3608.279811] __x64_sys_openat+0x24/0x30 [ 3608.282750] do_syscall_64+0xc1/0x440 [ 3608.287137] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3608.293502] RIP: 0033:0x7f1a63983332 [ 3608.296642] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3608.315104] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3608.317991] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3608.323977] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3608.333174] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3608.335595] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3608.340700] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3612.648071] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1589s! [ 3612.653714] Showing busy workqueues and worker pools: [ 3612.663215] workqueue events: flags=0x0 [ 3612.665496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3612.668592] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3612.671879] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3612.674662] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3612.679127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3612.682064] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3612.686681] workqueue events_long: flags=0x0 [ 3612.688991] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3612.692238] pending: lu_objects_flush [obdclass] [ 3612.695119] workqueue events_unbound: flags=0x2 [ 3612.697366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3612.700706] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3612.706081] workqueue events_power_efficient: flags=0x80 [ 3612.708634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3612.711747] pending: neigh_periodic_work, do_cache_clean [ 3612.714586] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3612.717874] pending: neigh_periodic_work, check_lifetime [ 3612.720423] workqueue mm_percpu_wq: flags=0x8 [ 3612.722330] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3612.725356] pending: vmstat_update [ 3612.727190] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3612.730229] pending: vmstat_update [ 3612.731972] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3612.735163] pending: vmstat_update [ 3612.736978] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3612.740021] pending: vmstat_update [ 3612.741691] workqueue writeback: flags=0x4a [ 3612.743664] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3612.747099] pending: wb_workfn, wb_workfn [ 3612.749097] workqueue xprtiod: flags=0xa [ 3612.751017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3612.754004] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3612.757622] workqueue ipv6_addrconf: flags=0x40008 [ 3612.760047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3612.763176] pending: addrconf_verify_work [ 3612.765318] workqueue knbd0-recv: flags=0x1a [ 3612.767155] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3612.770205] in-flight: 647:recv_work [ 3612.772123] workqueue ptlrpc_pinger: flags=0x2 [ 3612.774208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3612.777170] pending: ptlrpc_pinger_main [ptlrpc] [ 3612.779852] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3602s workers=2 idle: 652 [ 3620.057075] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3620.062154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3620.066946] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3620.084567] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3620.084582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3620.084586] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3620.084600] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3620.084604] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3620.084610] RAX: 000000000000270c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3620.084612] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3620.084614] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3620.084616] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3620.084618] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3620.084621] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3620.084624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3620.084627] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3620.084636] Call Trace: [ 3620.084641] [ 3620.084647] ? show_regs.cold.9+0x22/0x2f [ 3620.084658] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3620.084664] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3620.084669] ? watchdog+0x40/0x40 [ 3620.084674] ? hrtimer_interrupt+0x128/0x2c0 [ 3620.084678] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3620.084686] ? apic_timer_interrupt+0xf/0x20 [ 3620.084691] [ 3620.084693] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3620.084701] do_raw_spin_lock+0x177/0x190 [ 3620.084706] _raw_spin_lock+0x21/0x30 [ 3620.084712] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3620.085255] ? do_raw_spin_unlock+0x75/0x190 [ 3620.085261] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3620.085458] ? do_wait_intr+0xf0/0xf0 [ 3620.085466] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3620.085786] mdc_close+0x53b/0xd70 [mdc] [ 3620.085907] lmv_close+0x18d/0x490 [lmv] [ 3620.085942] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3620.086065] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3620.086154] ll_md_close+0x274/0x800 [lustre] [ 3620.086774] ll_file_release+0x128/0xc50 [lustre] [ 3620.086932] __fput+0x10d/0x440 [ 3620.086942] ____fput+0x12/0x20 [ 3620.087085] task_work_run+0xc6/0x110 [ 3620.087093] exit_to_usermode_loop+0x1dd/0x1f0 [ 3620.087100] do_syscall_64+0x3d6/0x440 [ 3620.087105] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3620.087113] RIP: 0033:0x7f1a63983cd8 [ 3620.087122] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3620.087126] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3620.087130] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3620.087133] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3620.087135] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3620.087137] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3620.087139] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3620.277024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3620.299679] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3620.306588] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3620.311279] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3620.314676] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3620.328339] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3620.333461] RAX: 000000000000002e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3620.337300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3620.341518] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3620.345151] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3620.349417] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3620.353764] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3620.357913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3620.361683] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3620.365384] Call Trace: [ 3620.366906] [ 3620.367939] ? show_regs.cold.9+0x22/0x2f [ 3620.371807] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3620.375043] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3620.378375] ? watchdog+0x40/0x40 [ 3620.379978] ? hrtimer_interrupt+0x128/0x2c0 [ 3620.382220] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3620.384757] ? apic_timer_interrupt+0xf/0x20 [ 3620.387306] [ 3620.388444] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3620.391914] do_raw_spin_lock+0x177/0x190 [ 3620.394052] _raw_spin_lock+0x21/0x30 [ 3620.396048] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3620.401097] ? do_raw_spin_unlock+0x75/0x190 [ 3620.403593] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3620.406467] ? do_wait_intr+0xf0/0xf0 [ 3620.408484] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3620.411728] mdc_close+0x53b/0xd70 [mdc] [ 3620.413933] lmv_close+0x18d/0x490 [lmv] [ 3620.415852] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3620.418996] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3620.421795] ll_md_close+0x274/0x800 [lustre] [ 3620.424645] ll_file_release+0x128/0xc50 [lustre] [ 3620.427786] __fput+0x10d/0x440 [ 3620.429486] ____fput+0x12/0x20 [ 3620.432527] task_work_run+0xc6/0x110 [ 3620.434434] exit_to_usermode_loop+0x1dd/0x1f0 [ 3620.436783] do_syscall_64+0x3d6/0x440 [ 3620.438486] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3620.441146] RIP: 0033:0x7f1a63983cd8 [ 3620.443113] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3620.452026] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3620.455776] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3620.463382] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3620.471651] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3620.478557] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3620.486478] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3632.061023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3632.070499] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3632.094073] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3632.102342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3632.109258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3632.112636] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3632.123194] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3632.127524] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3632.135142] RDX: 0000000000001e86 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3632.140212] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3632.145583] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3632.152855] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3632.158727] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3632.165221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3632.168838] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3632.175255] Call Trace: [ 3632.176257] [ 3632.177589] ? show_regs.cold.9+0x22/0x2f [ 3632.180204] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3632.183674] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3632.185621] ? watchdog+0x40/0x40 [ 3632.187637] ? hrtimer_interrupt+0x128/0x2c0 [ 3632.190599] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3632.193291] ? apic_timer_interrupt+0xf/0x20 [ 3632.197698] [ 3632.199087] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3632.203275] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3632.213330] do_raw_spin_lock+0x177/0x190 [ 3632.217161] _raw_spin_lock+0x21/0x30 [ 3632.219203] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3632.223105] obd_mod_free+0x46/0x140 [ptlrpc] [ 3632.226630] mdc_commit_open+0x91/0xb0 [mdc] [ 3632.229285] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3632.235197] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3632.239867] after_reply+0xe2c/0x1780 [ptlrpc] [ 3632.242836] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3632.245461] ? __next_timer_interrupt+0x160/0x160 [ 3632.251984] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3632.255183] ? do_wait_intr+0xf0/0xf0 [ 3632.257298] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3632.262965] kthread+0x1d1/0x200 [ 3632.265342] ? set_kthread_struct+0x70/0x70 [ 3632.267320] ret_from_fork+0x1f/0x30 [ 3640.058181] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3640.065019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3640.110449] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3640.122079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3640.128822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3640.134337] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3640.151556] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3640.158146] RAX: 0000000000006c05 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3640.163772] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3640.170036] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3640.180641] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3640.190225] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3640.198172] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3640.204387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3640.206437] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3640.212578] Call Trace: [ 3640.215462] [ 3640.222511] ? show_regs.cold.9+0x22/0x2f [ 3640.228719] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3640.232924] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3640.239021] ? watchdog+0x40/0x40 [ 3640.241037] ? hrtimer_interrupt+0x128/0x2c0 [ 3640.243727] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3640.246119] ? apic_timer_interrupt+0xf/0x20 [ 3640.248686] [ 3640.250178] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3640.253277] do_raw_spin_lock+0x177/0x190 [ 3640.256014] _raw_spin_lock+0x21/0x30 [ 3640.257335] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3640.260445] ? do_raw_spin_unlock+0x75/0x190 [ 3640.262537] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3640.265604] ? do_wait_intr+0xf0/0xf0 [ 3640.267005] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3640.274170] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3640.281731] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3640.290583] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3640.295892] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3640.299158] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3640.309425] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3640.316353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3640.320257] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3640.334602] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3640.336788] lmv_intent_open+0x338/0x1150 [lmv] [ 3640.348598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3640.359400] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3640.361459] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3640.368517] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3640.372972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3640.377892] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3640.381703] ? do_raw_spin_unlock+0x75/0x190 [ 3640.383215] ? _raw_spin_unlock+0x12/0x30 [ 3640.386399] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3640.391741] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3640.398161] ll_atomic_open+0x36a/0xd70 [lustre] [ 3640.406106] lookup_open+0x356/0xb40 [ 3640.410178] do_last+0x313/0xfc0 [ 3640.415247] ? nd_jump_root+0xe5/0x160 [ 3640.418809] ? path_init+0x437/0x520 [ 3640.420948] path_openat+0xf7/0x500 [ 3640.424065] ? do_raw_spin_unlock+0x75/0x190 [ 3640.428841] do_filp_open+0x99/0x140 [ 3640.434368] ? getname_flags+0x6e/0x330 [ 3640.438873] ? __check_object_size+0xff/0x256 [ 3640.444427] ? do_raw_spin_unlock+0x75/0x190 [ 3640.446449] ? _raw_spin_unlock+0x12/0x30 [ 3640.456250] do_sys_openat2+0x2b4/0x410 [ 3640.463326] do_sys_open+0x73/0xa0 [ 3640.465249] __x64_sys_openat+0x24/0x30 [ 3640.467599] do_syscall_64+0xc1/0x440 [ 3640.473035] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3640.479273] RIP: 0033:0x7f1a63983332 [ 3640.481247] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3640.500787] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3640.510994] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3640.516079] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3640.520064] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3640.525164] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3640.528568] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3642.848981] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1619s! [ 3642.854687] Showing busy workqueues and worker pools: [ 3642.866296] workqueue events: flags=0x0 [ 3642.869808] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3642.875848] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3642.881191] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3642.884334] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3642.890723] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3642.895185] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3642.906954] workqueue events_long: flags=0x0 [ 3642.916110] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3642.925362] pending: lu_objects_flush [obdclass] [ 3642.935540] workqueue events_unbound: flags=0x2 [ 3642.941371] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3642.949433] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3642.960873] workqueue events_power_efficient: flags=0x80 [ 3642.965227] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3642.978942] pending: neigh_periodic_work, do_cache_clean [ 3642.990573] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3643.003725] pending: neigh_periodic_work, check_lifetime [ 3643.007033] workqueue mm_percpu_wq: flags=0x8 [ 3643.012336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3643.032703] pending: vmstat_update [ 3643.034125] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3643.048444] pending: vmstat_update [ 3643.050238] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3643.054792] pending: vmstat_update [ 3643.056313] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3643.060486] pending: vmstat_update [ 3643.063264] workqueue writeback: flags=0x4a [ 3643.064937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3643.072414] pending: wb_workfn, wb_workfn [ 3643.086168] workqueue xprtiod: flags=0xa [ 3643.090912] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3643.098559] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3643.105545] workqueue ipv6_addrconf: flags=0x40008 [ 3643.110912] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3643.120410] pending: addrconf_verify_work [ 3643.125179] workqueue knbd0-recv: flags=0x1a [ 3643.131504] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3643.139907] in-flight: 647:recv_work [ 3643.142825] workqueue ptlrpc_pinger: flags=0x2 [ 3643.147357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3643.153716] pending: ptlrpc_pinger_main [ptlrpc] [ 3643.159444] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3633s workers=2 idle: 652 [ 3648.056275] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3648.057085] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3648.059414] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3648.059495] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3648.059503] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3648.059506] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3648.059522] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3648.059526] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3648.059530] RAX: 0000000000002d96 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3648.059532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3648.059534] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3648.059536] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3648.059538] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3648.059542] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3648.059544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3648.059546] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3648.059564] Call Trace: [ 3648.059569] [ 3648.059575] ? show_regs.cold.9+0x22/0x2f [ 3648.059586] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3648.063678] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3648.063752] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3648.063760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3648.063763] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3648.063776] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3648.063781] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3648.063786] RAX: 0000000000001c50 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3648.063788] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3648.063790] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3648.063792] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3648.063794] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3648.063797] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3648.063799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3648.063801] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3648.063811] Call Trace: [ 3648.063816] [ 3648.063821] ? show_regs.cold.9+0x22/0x2f [ 3648.063832] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3648.063838] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3648.063842] ? watchdog+0x40/0x40 [ 3648.063847] ? hrtimer_interrupt+0x128/0x2c0 [ 3648.063851] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3648.063859] ? apic_timer_interrupt+0xf/0x20 [ 3648.063863] [ 3648.063866] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3648.063872] do_raw_spin_lock+0x177/0x190 [ 3648.063877] _raw_spin_lock+0x21/0x30 [ 3648.063881] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3648.064667] ? do_raw_spin_unlock+0x75/0x190 [ 3648.087157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3648.087175] ? watchdog+0x40/0x40 [ 3648.087181] ? hrtimer_interrupt+0x128/0x2c0 [ 3648.087186] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3648.087193] ? apic_timer_interrupt+0xf/0x20 [ 3648.087198] [ 3648.087202] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3648.087210] do_raw_spin_lock+0x177/0x190 [ 3648.087216] _raw_spin_lock+0x21/0x30 [ 3648.087221] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3648.087922] ? do_raw_spin_unlock+0x75/0x190 [ 3648.087929] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3648.094387] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3648.094848] ? do_wait_intr+0xf0/0xf0 [ 3648.094858] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3648.095138] mdc_close+0x53b/0xd70 [mdc] [ 3648.095176] lmv_close+0x18d/0x490 [lmv] [ 3648.095215] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3648.095402] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3648.095495] ll_md_close+0x274/0x800 [lustre] [ 3648.095657] ll_file_release+0x128/0xc50 [lustre] [ 3648.095741] __fput+0x10d/0x440 [ 3648.095750] ____fput+0x12/0x20 [ 3648.095754] task_work_run+0xc6/0x110 [ 3648.095762] exit_to_usermode_loop+0x1dd/0x1f0 [ 3648.095774] do_syscall_64+0x3d6/0x440 [ 3648.095778] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3648.095786] RIP: 0033:0x7f1a63983cd8 [ 3648.095794] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3648.095798] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3648.095803] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3648.095805] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3648.095807] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3648.095810] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3648.095812] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3648.478437] ? do_wait_intr+0xf0/0xf0 [ 3648.480915] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3648.484499] mdc_close+0x53b/0xd70 [mdc] [ 3648.487106] lmv_close+0x18d/0x490 [lmv] [ 3648.492490] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3648.504633] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3648.511207] ll_md_close+0x274/0x800 [lustre] [ 3648.515028] ll_file_release+0x128/0xc50 [lustre] [ 3648.528219] __fput+0x10d/0x440 [ 3648.535287] ____fput+0x12/0x20 [ 3648.538986] task_work_run+0xc6/0x110 [ 3648.544861] exit_to_usermode_loop+0x1dd/0x1f0 [ 3648.546751] do_syscall_64+0x3d6/0x440 [ 3648.548666] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3648.554680] RIP: 0033:0x7f1a63983cd8 [ 3648.555905] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3648.572434] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3648.580653] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3648.594848] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3648.600883] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3648.606684] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3648.613825] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3660.063360] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3660.072151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3660.124680] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3660.135228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3660.147889] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3660.157700] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3660.180932] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3660.186613] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3660.190811] RDX: 00000000000078b3 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3660.194946] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3660.198516] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3660.201244] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3660.205272] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3660.213841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3660.216948] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3660.223221] Call Trace: [ 3660.226601] [ 3660.227078] ? show_regs.cold.9+0x22/0x2f [ 3660.228710] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3660.231647] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3660.233885] ? watchdog+0x40/0x40 [ 3660.235319] ? hrtimer_interrupt+0x128/0x2c0 [ 3660.237153] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3660.240042] ? apic_timer_interrupt+0xf/0x20 [ 3660.242551] [ 3660.246231] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3660.249019] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3660.252073] do_raw_spin_lock+0x177/0x190 [ 3660.254372] _raw_spin_lock+0x21/0x30 [ 3660.256492] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3660.258986] obd_mod_free+0x46/0x140 [ptlrpc] [ 3660.262924] mdc_commit_open+0x91/0xb0 [mdc] [ 3660.264788] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3660.273412] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3660.281702] after_reply+0xe2c/0x1780 [ptlrpc] [ 3660.290343] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3660.303179] ? __next_timer_interrupt+0x160/0x160 [ 3660.313001] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3660.316090] ? do_wait_intr+0xf0/0xf0 [ 3660.318955] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3660.325361] kthread+0x1d1/0x200 [ 3660.328090] ? set_kthread_struct+0x70/0x70 [ 3660.330727] ret_from_fork+0x1f/0x30 [ 3668.053297] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3668.057468] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3668.100075] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3668.105695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3668.111118] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3668.113579] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3668.134430] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3668.142626] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3668.148339] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3668.152672] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3668.158230] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3668.162885] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3668.169653] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3668.180636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3668.184198] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3668.189621] Call Trace: [ 3668.190596] [ 3668.191319] ? show_regs.cold.9+0x22/0x2f [ 3668.197445] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3668.200791] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3668.205503] ? watchdog+0x40/0x40 [ 3668.206788] ? hrtimer_interrupt+0x128/0x2c0 [ 3668.209399] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3668.211175] ? apic_timer_interrupt+0xf/0x20 [ 3668.213958] [ 3668.215080] ? native_safe_halt+0x2e/0x50 [ 3668.218956] kvm_wait+0xcd/0xe0 [ 3668.220148] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3668.223222] do_raw_spin_lock+0x177/0x190 [ 3668.226199] _raw_spin_lock+0x21/0x30 [ 3668.230107] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3668.236742] ? do_raw_spin_unlock+0x75/0x190 [ 3668.239987] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3668.245199] ? do_wait_intr+0xf0/0xf0 [ 3668.248103] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3668.251426] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3668.258153] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3668.264764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3668.268151] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3668.273336] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3668.276883] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3668.284325] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3668.288085] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3668.298791] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3668.303070] lmv_intent_open+0x338/0x1150 [lmv] [ 3668.305797] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3668.309784] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3668.314299] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3668.319881] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3668.328080] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3668.337159] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3668.340149] ? do_raw_spin_unlock+0x75/0x190 [ 3668.342942] ? _raw_spin_unlock+0x12/0x30 [ 3668.345059] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3668.350179] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3668.355125] ll_atomic_open+0x36a/0xd70 [lustre] [ 3668.358682] lookup_open+0x356/0xb40 [ 3668.362180] do_last+0x313/0xfc0 [ 3668.364775] ? nd_jump_root+0xe5/0x160 [ 3668.367145] ? path_init+0x437/0x520 [ 3668.368479] path_openat+0xf7/0x500 [ 3668.370331] ? do_raw_spin_unlock+0x75/0x190 [ 3668.375980] do_filp_open+0x99/0x140 [ 3668.377784] ? getname_flags+0x6e/0x330 [ 3668.381185] ? __check_object_size+0xff/0x256 [ 3668.385287] ? do_raw_spin_unlock+0x75/0x190 [ 3668.390191] ? _raw_spin_unlock+0x12/0x30 [ 3668.393196] do_sys_openat2+0x2b4/0x410 [ 3668.397094] do_sys_open+0x73/0xa0 [ 3668.399832] __x64_sys_openat+0x24/0x30 [ 3668.401912] do_syscall_64+0xc1/0x440 [ 3668.403316] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3668.411254] RIP: 0033:0x7f1a63983332 [ 3668.416456] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3668.431530] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3668.434315] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3668.439181] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3668.446086] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3668.453093] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3668.457431] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3673.570333] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1650s! [ 3673.580977] Showing busy workqueues and worker pools: [ 3673.583052] workqueue events: flags=0x0 [ 3673.590538] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3673.595701] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3673.602457] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3673.606431] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3673.611991] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3673.615158] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3673.621530] workqueue events_long: flags=0x0 [ 3673.623153] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3673.626111] pending: lu_objects_flush [obdclass] [ 3673.628961] workqueue events_unbound: flags=0x2 [ 3673.630938] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3673.633805] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3673.640914] workqueue events_power_efficient: flags=0x80 [ 3673.647538] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3673.653475] pending: neigh_periodic_work, do_cache_clean [ 3673.655689] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3673.661643] pending: neigh_periodic_work, check_lifetime [ 3673.664439] workqueue mm_percpu_wq: flags=0x8 [ 3673.668145] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3673.672001] pending: vmstat_update [ 3673.673663] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3673.678656] pending: vmstat_update [ 3673.681240] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3673.687108] pending: vmstat_update [ 3673.689061] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3673.694879] pending: vmstat_update [ 3673.696776] workqueue writeback: flags=0x4a [ 3673.698915] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3673.701095] pending: wb_workfn, wb_workfn [ 3673.705835] workqueue xprtiod: flags=0xa [ 3673.707335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3673.709266] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3673.716651] workqueue ipv6_addrconf: flags=0x40008 [ 3673.722018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3673.727975] pending: addrconf_verify_work [ 3673.730793] workqueue knbd0-recv: flags=0x1a [ 3673.733370] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3673.736821] in-flight: 647:recv_work [ 3673.739053] workqueue ptlrpc_pinger: flags=0x2 [ 3673.741589] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3673.744570] pending: ptlrpc_pinger_main [ptlrpc] [ 3673.747373] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3663s workers=2 idle: 652 [ 3676.056961] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3676.057272] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3676.063967] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 3676.068855] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3676.068964] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3676.068972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3676.068975] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3676.068988] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3676.068992] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3676.068996] RAX: 0000000000000f53 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3676.068998] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3676.069000] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3676.069002] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3676.069050] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3676.069057] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3676.069061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3676.069065] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3676.069073] Call Trace: [ 3676.069079] [ 3676.069084] ? show_regs.cold.9+0x22/0x2f [ 3676.069094] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3676.069100] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3676.069116] ? watchdog+0x40/0x40 [ 3676.069124] ? hrtimer_interrupt+0x128/0x2c0 [ 3676.069128] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3676.069136] ? apic_timer_interrupt+0xf/0x20 [ 3676.069142] [ 3676.069149] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3676.069158] do_raw_spin_lock+0x177/0x190 [ 3676.069163] _raw_spin_lock+0x21/0x30 [ 3676.069168] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3676.069804] ? do_raw_spin_unlock+0x75/0x190 [ 3676.069923] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3676.070150] ? do_wait_intr+0xf0/0xf0 [ 3676.070158] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3676.070699] mdc_close+0x53b/0xd70 [mdc] [ 3676.070738] lmv_close+0x18d/0x490 [lmv] [ 3676.070770] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3676.070875] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3676.070956] ll_md_close+0x274/0x800 [lustre] [ 3676.071036] ll_file_release+0x128/0xc50 [lustre] [ 3676.071264] __fput+0x10d/0x440 [ 3676.071289] ____fput+0x12/0x20 [ 3676.071294] task_work_run+0xc6/0x110 [ 3676.071301] exit_to_usermode_loop+0x1dd/0x1f0 [ 3676.071309] do_syscall_64+0x3d6/0x440 [ 3676.071319] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3676.071331] RIP: 0033:0x7f1a63983cd8 [ 3676.071342] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3676.071345] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3676.071350] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3676.071353] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3676.071355] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3676.071357] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3676.071359] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3676.352868] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3676.382659] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3676.390478] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3676.399480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3676.406396] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3676.421096] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3676.426567] RAX: 000000000000570a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3676.429063] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3676.432731] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3676.436908] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3676.441824] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3676.448911] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3676.452793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3676.456482] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3676.462959] Call Trace: [ 3676.464436] [ 3676.465087] ? show_regs.cold.9+0x22/0x2f [ 3676.476684] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3676.481250] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3676.482963] ? watchdog+0x40/0x40 [ 3676.484707] ? hrtimer_interrupt+0x128/0x2c0 [ 3676.486518] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3676.488560] ? apic_timer_interrupt+0xf/0x20 [ 3676.491671] [ 3676.493043] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3676.495663] do_raw_spin_lock+0x177/0x190 [ 3676.497675] _raw_spin_lock+0x21/0x30 [ 3676.499000] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3676.501533] ? do_raw_spin_unlock+0x75/0x190 [ 3676.502946] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3676.505260] ? do_wait_intr+0xf0/0xf0 [ 3676.506607] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3676.508898] mdc_close+0x53b/0xd70 [mdc] [ 3676.510722] lmv_close+0x18d/0x490 [lmv] [ 3676.512378] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3676.514876] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3676.516987] ll_md_close+0x274/0x800 [lustre] [ 3676.519438] ll_file_release+0x128/0xc50 [lustre] [ 3676.521650] __fput+0x10d/0x440 [ 3676.522862] ____fput+0x12/0x20 [ 3676.524153] task_work_run+0xc6/0x110 [ 3676.526730] exit_to_usermode_loop+0x1dd/0x1f0 [ 3676.528985] do_syscall_64+0x3d6/0x440 [ 3676.530920] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3676.533786] RIP: 0033:0x7f1a63983cd8 [ 3676.535954] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3676.546506] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3676.549971] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3676.552976] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3676.556617] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3676.560036] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3676.563616] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3688.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3688.061472] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3688.082763] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3688.088015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3688.092232] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3688.094494] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3688.102943] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3688.106146] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3688.109409] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3688.112718] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3688.116234] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3688.122189] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3688.126507] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3688.130592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3688.136439] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3688.142862] Call Trace: [ 3688.143876] [ 3688.144745] ? show_regs.cold.9+0x22/0x2f [ 3688.146083] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3688.148155] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3688.151362] ? watchdog+0x40/0x40 [ 3688.152697] ? hrtimer_interrupt+0x128/0x2c0 [ 3688.154293] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3688.156805] ? apic_timer_interrupt+0xf/0x20 [ 3688.159758] [ 3688.160444] ? native_safe_halt+0x2e/0x50 [ 3688.161966] kvm_wait+0xcd/0xe0 [ 3688.163095] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3688.167689] do_raw_spin_lock+0x177/0x190 [ 3688.169180] _raw_spin_lock+0x21/0x30 [ 3688.171222] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3688.173742] obd_mod_free+0x46/0x140 [ptlrpc] [ 3688.177738] mdc_commit_open+0x91/0xb0 [mdc] [ 3688.182035] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3688.187632] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3688.190993] after_reply+0xe2c/0x1780 [ptlrpc] [ 3688.193554] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3688.196902] ? __next_timer_interrupt+0x160/0x160 [ 3688.199662] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3688.202668] ? do_wait_intr+0xf0/0xf0 [ 3688.204540] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3688.207098] kthread+0x1d1/0x200 [ 3688.208982] ? set_kthread_struct+0x70/0x70 [ 3688.211031] ret_from_fork+0x1f/0x30 [ 3700.053187] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3700.058925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3700.095008] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3700.102887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3700.111241] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3700.121488] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3700.139252] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3700.144369] RAX: 0000000000007d83 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3700.148029] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3700.161923] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3700.167699] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3700.170843] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3700.173874] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3700.180635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3700.184302] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3700.188521] Call Trace: [ 3700.189533] [ 3700.190356] ? show_regs.cold.9+0x22/0x2f [ 3700.196640] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3700.199560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3700.204842] ? watchdog+0x40/0x40 [ 3700.208951] ? hrtimer_interrupt+0x128/0x2c0 [ 3700.214508] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3700.217997] ? apic_timer_interrupt+0xf/0x20 [ 3700.221477] [ 3700.222654] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3700.225478] do_raw_spin_lock+0x177/0x190 [ 3700.227532] _raw_spin_lock+0x21/0x30 [ 3700.229387] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3700.232663] ? do_raw_spin_unlock+0x75/0x190 [ 3700.237271] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3700.241018] ? do_wait_intr+0xf0/0xf0 [ 3700.248573] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3700.251836] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3700.254784] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3700.261745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3700.265527] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3700.271583] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3700.273541] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3700.277234] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3700.279510] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3700.286318] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3700.289916] lmv_intent_open+0x338/0x1150 [lmv] [ 3700.291230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3700.297969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3700.301001] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3700.304609] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3700.307954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3700.311587] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3700.315379] ? do_raw_spin_unlock+0x75/0x190 [ 3700.317371] ? _raw_spin_unlock+0x12/0x30 [ 3700.321899] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3700.326501] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3700.330854] ll_atomic_open+0x36a/0xd70 [lustre] [ 3700.333221] lookup_open+0x356/0xb40 [ 3700.339483] do_last+0x313/0xfc0 [ 3700.341633] ? nd_jump_root+0xe5/0x160 [ 3700.344365] ? path_init+0x437/0x520 [ 3700.352963] path_openat+0xf7/0x500 [ 3700.356751] ? do_raw_spin_unlock+0x75/0x190 [ 3700.359911] do_filp_open+0x99/0x140 [ 3700.363279] ? getname_flags+0x6e/0x330 [ 3700.367447] ? __check_object_size+0xff/0x256 [ 3700.370349] ? do_raw_spin_unlock+0x75/0x190 [ 3700.375097] ? _raw_spin_unlock+0x12/0x30 [ 3700.376984] do_sys_openat2+0x2b4/0x410 [ 3700.381206] do_sys_open+0x73/0xa0 [ 3700.383511] __x64_sys_openat+0x24/0x30 [ 3700.386724] do_syscall_64+0xc1/0x440 [ 3700.388694] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3700.393546] RIP: 0033:0x7f1a63983332 [ 3700.395921] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3700.408774] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3700.414904] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3700.417967] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3700.421238] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3700.423262] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3700.427262] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3702.446014] rcu: INFO: rcu_sched self-detected stall on CPU [ 3702.447752] rcu: 3-...!: (989915 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 3702.450696] (t=1680044 jiffies g=269633 q=760) [ 3702.451976] rcu: rcu_sched kthread starved for 1679691 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3702.454762] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3702.457196] rcu: RCU grace-period kthread stack dump: [ 3702.458610] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3702.460948] Call Trace: [ 3702.461682] __schedule+0x351/0xcb0 [ 3702.462688] schedule+0xc0/0x180 [ 3702.463785] schedule_timeout+0xb4/0x190 [ 3702.464866] ? do_raw_spin_unlock+0x75/0x190 [ 3702.466125] ? __next_timer_interrupt+0x160/0x160 [ 3702.467365] ? prepare_to_swait_event+0xf3/0x140 [ 3702.468597] rcu_gp_kthread+0x8dd/0x15b0 [ 3702.469657] ? force_qs_rnp+0x2c0/0x2c0 [ 3702.470710] kthread+0x1d1/0x200 [ 3702.471616] ? set_kthread_struct+0x70/0x70 [ 3702.472686] ret_from_fork+0x1f/0x30 [ 3702.473684] rcu: Stack dump where RCU GP kthread last ran: [ 3702.475195] Sending NMI from CPU 3 to CPUs 2: [ 3702.476423] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 3702.477378] NMI backtrace for cpu 3 [ 3702.480641] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3702.483638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3702.486393] Call Trace: [ 3702.487135] [ 3702.487660] ? dump_stack+0xbb/0x10e [ 3702.488814] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3702.490592] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3702.492267] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3702.493958] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3702.495808] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3702.497426] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3702.498789] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3702.500854] ? update_process_times+0x6d/0xc0 [ 3702.502403] ? tick_sched_handle.isra.18+0x27/0x90 [ 3702.503976] ? tick_sched_timer+0x53/0xb0 [ 3702.505661] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3702.507212] ? tick_sched_handle.isra.18+0x90/0x90 [ 3702.508458] ? hrtimer_interrupt+0x128/0x2c0 [ 3702.509721] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3702.511761] ? apic_timer_interrupt+0xf/0x20 [ 3702.514599] [ 3702.515682] ? native_safe_halt+0x2e/0x50 [ 3702.517812] ? kvm_wait+0xcd/0xe0 [ 3702.518887] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3702.520760] ? do_raw_spin_lock+0x177/0x190 [ 3702.522508] ? _raw_spin_lock+0x21/0x30 [ 3702.523685] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3702.526682] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3702.529296] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3702.530633] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3702.533298] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3702.537926] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3702.539527] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3702.541157] ? __next_timer_interrupt+0x160/0x160 [ 3702.542348] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3702.543688] ? do_wait_intr+0xf0/0xf0 [ 3702.544673] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3702.548903] ? kthread+0x1d1/0x200 [ 3702.550685] ? set_kthread_struct+0x70/0x70 [ 3702.551989] ? ret_from_fork+0x1f/0x30 [ 3703.778492] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1680s! [ 3703.798153] Showing busy workqueues and worker pools: [ 3703.800024] workqueue events: flags=0x0 [ 3703.801691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3703.804369] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3703.808905] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3703.813493] pending: sec_gc_main [ptlrpc] [ 3703.815867] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3703.819331] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3703.825196] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3703.828116] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3703.833510] workqueue events_long: flags=0x0 [ 3703.835254] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3703.837911] pending: lu_objects_flush [obdclass] [ 3703.840235] workqueue events_unbound: flags=0x2 [ 3703.842311] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3703.844743] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3703.849598] workqueue events_power_efficient: flags=0x80 [ 3703.852639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3703.855366] pending: neigh_periodic_work, do_cache_clean [ 3703.857564] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3703.859870] pending: neigh_periodic_work, check_lifetime [ 3703.862609] workqueue mm_percpu_wq: flags=0x8 [ 3703.864540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3703.868200] pending: vmstat_update [ 3703.869905] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3703.872922] pending: vmstat_update [ 3703.874593] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3703.877692] pending: vmstat_update [ 3703.879262] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3703.884485] pending: vmstat_update [ 3703.885967] workqueue writeback: flags=0x4a [ 3703.886939] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3703.890106] pending: wb_workfn, wb_workfn [ 3703.895341] workqueue xprtiod: flags=0xa [ 3703.897576] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3703.899909] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3703.903772] workqueue ipv6_addrconf: flags=0x40008 [ 3703.905694] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3703.908387] pending: addrconf_verify_work [ 3703.910196] workqueue knbd0-recv: flags=0x1a [ 3703.912356] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3703.914983] in-flight: 647:recv_work [ 3703.916589] workqueue ptlrpc_pinger: flags=0x2 [ 3703.918423] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3703.920964] pending: ptlrpc_pinger_main [ptlrpc] [ 3703.923825] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3693s workers=2 idle: 652 [ 3704.056066] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3704.063488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3704.084213] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3704.090217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3704.094763] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3704.096971] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3704.105436] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3704.108637] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3704.111685] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3704.114871] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3704.121355] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3704.127556] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3704.131457] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3704.137879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3704.142300] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3704.147847] Call Trace: [ 3704.154410] [ 3704.157398] ? show_regs.cold.9+0x22/0x2f [ 3704.159833] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3704.163398] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3704.167776] ? watchdog+0x40/0x40 [ 3704.171322] ? hrtimer_interrupt+0x128/0x2c0 [ 3704.174668] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3704.179218] ? apic_timer_interrupt+0xf/0x20 [ 3704.180948] [ 3704.182609] ? native_safe_halt+0x2e/0x50 [ 3704.187195] kvm_wait+0xcd/0xe0 [ 3704.188523] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3704.190750] do_raw_spin_lock+0x177/0x190 [ 3704.192471] _raw_spin_lock+0x21/0x30 [ 3704.194528] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3704.197529] ? do_raw_spin_unlock+0x75/0x190 [ 3704.199305] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3704.202193] ? do_wait_intr+0xf0/0xf0 [ 3704.203704] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3704.205972] mdc_close+0x53b/0xd70 [mdc] [ 3704.207730] lmv_close+0x18d/0x490 [lmv] [ 3704.209599] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3704.212255] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3704.214468] ll_md_close+0x274/0x800 [lustre] [ 3704.218339] ll_file_release+0x128/0xc50 [lustre] [ 3704.220764] __fput+0x10d/0x440 [ 3704.223249] ____fput+0x12/0x20 [ 3704.224864] task_work_run+0xc6/0x110 [ 3704.225931] exit_to_usermode_loop+0x1dd/0x1f0 [ 3704.231762] do_syscall_64+0x3d6/0x440 [ 3704.235637] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3704.239322] RIP: 0033:0x7f1a63983cd8 [ 3704.240855] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3704.248677] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3704.251246] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3704.253800] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3704.256605] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3704.259510] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3704.261361] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3728.053370] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3728.058015] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3728.060248] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3728.064588] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 3728.064601] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3728.064675] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3728.064682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3728.064685] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3728.064698] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3728.064702] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3728.064707] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3728.064710] RDX: 00000000000077c3 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3728.064712] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3728.064714] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3728.064717] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3728.064720] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3728.064723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3728.064725] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3728.064734] Call Trace: [ 3728.064740] [ 3728.064745] ? show_regs.cold.9+0x22/0x2f [ 3728.064755] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3728.064761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3728.064765] ? watchdog+0x40/0x40 [ 3728.064784] ? hrtimer_interrupt+0x128/0x2c0 [ 3728.064790] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3728.064799] ? apic_timer_interrupt+0xf/0x20 [ 3728.064804] [ 3728.064807] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3728.064812] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3728.064818] do_raw_spin_lock+0x177/0x190 [ 3728.064823] _raw_spin_lock+0x21/0x30 [ 3728.064828] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3728.065374] obd_mod_free+0x46/0x140 [ptlrpc] [ 3728.065581] mdc_commit_open+0x91/0xb0 [mdc] [ 3728.065616] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3728.065970] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3728.066192] after_reply+0xe2c/0x1780 [ptlrpc] [ 3728.066572] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3728.066767] ? __next_timer_interrupt+0x160/0x160 [ 3728.066840] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3728.067163] ? do_wait_intr+0xf0/0xf0 [ 3728.067172] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3728.067435] kthread+0x1d1/0x200 [ 3728.067442] ? set_kthread_struct+0x70/0x70 [ 3728.067445] ret_from_fork+0x1f/0x30 [ 3728.096921] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3728.096935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3728.096938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3728.102599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3728.102681] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3728.102689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3728.102691] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3728.102707] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3728.102711] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3728.102715] RAX: 000000000000730e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3728.102718] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3728.102720] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3728.102722] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3728.102724] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3728.102728] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3728.102731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3728.102734] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3728.102742] Call Trace: [ 3728.102748] [ 3728.102754] ? show_regs.cold.9+0x22/0x2f [ 3728.105027] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3728.110161] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3728.110174] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3728.110180] RAX: 0000000000005b09 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3728.110183] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3728.110185] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3728.110187] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3728.110190] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3728.110193] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3728.110196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3728.110198] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3728.110207] Call Trace: [ 3728.110212] [ 3728.110217] ? show_regs.cold.9+0x22/0x2f [ 3728.110230] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3728.110236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3728.110240] ? watchdog+0x40/0x40 [ 3728.110244] ? hrtimer_interrupt+0x128/0x2c0 [ 3728.110247] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3728.110257] ? apic_timer_interrupt+0xf/0x20 [ 3728.110262] [ 3728.110265] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3728.149996] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3728.150014] ? watchdog+0x40/0x40 [ 3728.150019] ? hrtimer_interrupt+0x128/0x2c0 [ 3728.150024] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3728.150031] ? apic_timer_interrupt+0xf/0x20 [ 3728.150036] [ 3728.150155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3728.150168] do_raw_spin_lock+0x177/0x190 [ 3728.150174] _raw_spin_lock+0x21/0x30 [ 3728.150180] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3728.162780] do_raw_spin_lock+0x177/0x190 [ 3728.162799] _raw_spin_lock+0x21/0x30 [ 3728.162805] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3728.166038] ? do_raw_spin_unlock+0x75/0x190 [ 3728.166058] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3728.166438] ? do_wait_intr+0xf0/0xf0 [ 3728.166446] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3728.166857] mdc_close+0x53b/0xd70 [mdc] [ 3728.166974] lmv_close+0x18d/0x490 [lmv] [ 3728.167008] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3728.168835] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3728.169308] ll_md_close+0x274/0x800 [lustre] [ 3728.169399] ll_file_release+0x128/0xc50 [lustre] [ 3728.169487] __fput+0x10d/0x440 [ 3728.169588] ____fput+0x12/0x20 [ 3728.169596] task_work_run+0xc6/0x110 [ 3728.169608] exit_to_usermode_loop+0x1dd/0x1f0 [ 3728.169617] do_syscall_64+0x3d6/0x440 [ 3728.169622] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3728.169632] RIP: 0033:0x7f1a63983cd8 [ 3728.169641] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3728.169645] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3728.169651] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3728.169653] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3728.169656] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3728.169658] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3728.169660] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3728.535096] ? do_raw_spin_unlock+0x75/0x190 [ 3728.536574] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3728.538581] ? do_wait_intr+0xf0/0xf0 [ 3728.539689] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3728.541938] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3728.543877] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3728.546125] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3728.548240] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3728.550222] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3728.551497] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3728.552951] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3728.554703] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3728.557289] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3728.559896] lmv_intent_open+0x338/0x1150 [lmv] [ 3728.561939] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3728.564144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3728.566529] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3728.568387] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3728.570670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3728.573047] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3728.580209] ? do_raw_spin_unlock+0x75/0x190 [ 3728.583776] ? _raw_spin_unlock+0x12/0x30 [ 3728.585678] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3728.589427] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3728.593297] ll_atomic_open+0x36a/0xd70 [lustre] [ 3728.596291] lookup_open+0x356/0xb40 [ 3728.598272] do_last+0x313/0xfc0 [ 3728.603283] ? nd_jump_root+0xe5/0x160 [ 3728.606163] ? path_init+0x437/0x520 [ 3728.608305] path_openat+0xf7/0x500 [ 3728.610742] ? do_raw_spin_unlock+0x75/0x190 [ 3728.613573] do_filp_open+0x99/0x140 [ 3728.615551] ? getname_flags+0x6e/0x330 [ 3728.618172] ? __check_object_size+0xff/0x256 [ 3728.620878] ? do_raw_spin_unlock+0x75/0x190 [ 3728.623336] ? _raw_spin_unlock+0x12/0x30 [ 3728.626261] do_sys_openat2+0x2b4/0x410 [ 3728.628554] do_sys_open+0x73/0xa0 [ 3728.630427] __x64_sys_openat+0x24/0x30 [ 3728.632695] do_syscall_64+0xc1/0x440 [ 3728.637890] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3728.643538] RIP: 0033:0x7f1a63983332 [ 3728.649550] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3728.669053] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3728.674749] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3728.682437] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3728.694715] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3728.703580] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3728.717143] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3732.056009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3732.066718] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3732.090651] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3732.095874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3732.099953] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3732.102959] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3732.113716] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3732.117539] RAX: 0000000000002ca3 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3732.121078] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3732.125040] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3732.127694] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3732.130696] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3732.133670] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3732.136953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3732.139518] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3732.142140] Call Trace: [ 3732.143023] [ 3732.143910] ? show_regs.cold.9+0x22/0x2f [ 3732.145499] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3732.146960] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3732.148845] ? watchdog+0x40/0x40 [ 3732.150330] ? hrtimer_interrupt+0x128/0x2c0 [ 3732.154853] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3732.157391] ? apic_timer_interrupt+0xf/0x20 [ 3732.161647] [ 3732.162875] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3732.168405] do_raw_spin_lock+0x177/0x190 [ 3732.171307] _raw_spin_lock+0x21/0x30 [ 3732.176849] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3732.179856] ? do_raw_spin_unlock+0x75/0x190 [ 3732.181662] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3732.184238] ? do_wait_intr+0xf0/0xf0 [ 3732.186570] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3732.190605] mdc_close+0x53b/0xd70 [mdc] [ 3732.193010] lmv_close+0x18d/0x490 [lmv] [ 3732.198998] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3732.203799] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3732.205944] ll_md_close+0x274/0x800 [lustre] [ 3732.208023] ll_file_release+0x128/0xc50 [lustre] [ 3732.210689] __fput+0x10d/0x440 [ 3732.212671] ____fput+0x12/0x20 [ 3732.214571] task_work_run+0xc6/0x110 [ 3732.215956] exit_to_usermode_loop+0x1dd/0x1f0 [ 3732.226655] do_syscall_64+0x3d6/0x440 [ 3732.233420] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3732.237607] RIP: 0033:0x7f1a63983cd8 [ 3732.239299] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3732.264876] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3732.272013] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3732.276513] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3732.283025] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3732.287010] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3732.290675] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3733.989968] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1710s! [ 3734.003554] Showing busy workqueues and worker pools: [ 3734.006516] workqueue events: flags=0x0 [ 3734.010280] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3734.020450] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3734.026656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3734.033870] pending: sec_gc_main [ptlrpc] [ 3734.040792] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3734.048090] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3734.065145] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3734.074440] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3734.094443] workqueue events_long: flags=0x0 [ 3734.108795] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3734.118326] pending: lu_objects_flush [obdclass] [ 3734.132803] workqueue events_unbound: flags=0x2 [ 3734.139577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3734.151423] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3734.167563] workqueue events_power_efficient: flags=0x80 [ 3734.177021] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3734.188395] pending: neigh_periodic_work, do_cache_clean [ 3734.195509] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3734.203064] pending: neigh_periodic_work, check_lifetime [ 3734.210320] workqueue mm_percpu_wq: flags=0x8 [ 3734.214499] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3734.228627] pending: vmstat_update [ 3734.233171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3734.240874] pending: vmstat_update [ 3734.245820] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3734.256764] pending: vmstat_update [ 3734.259093] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3734.268468] pending: vmstat_update [ 3734.275896] workqueue writeback: flags=0x4a [ 3734.278194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3734.282682] pending: wb_workfn, wb_workfn [ 3734.285677] workqueue xprtiod: flags=0xa [ 3734.288228] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3734.292314] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3734.299319] workqueue ipv6_addrconf: flags=0x40008 [ 3734.302170] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3734.307698] pending: addrconf_verify_work [ 3734.314032] workqueue knbd0-recv: flags=0x1a [ 3734.316319] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3734.319851] in-flight: 647:recv_work [ 3734.321877] workqueue ptlrpc_pinger: flags=0x2 [ 3734.325290] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3734.330727] pending: ptlrpc_pinger_main [ptlrpc] [ 3734.335465] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3724s workers=2 idle: 652 [ 3756.058035] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 3756.058096] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3756.066812] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3756.066965] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3756.066972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3756.066975] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3756.066989] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3756.066993] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3756.066998] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3756.067000] RDX: 0000000000007e8a RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3756.067002] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3756.067004] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3756.067006] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3756.067010] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3756.067012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3756.067015] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3756.067023] Call Trace: [ 3756.067029] [ 3756.067035] ? show_regs.cold.9+0x22/0x2f [ 3756.067046] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3756.067053] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3756.067057] ? watchdog+0x40/0x40 [ 3756.067061] ? hrtimer_interrupt+0x128/0x2c0 [ 3756.067065] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3756.067072] ? apic_timer_interrupt+0xf/0x20 [ 3756.067076] [ 3756.067079] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3756.067083] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3756.067087] do_raw_spin_lock+0x177/0x190 [ 3756.067091] _raw_spin_lock+0x21/0x30 [ 3756.067096] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3756.067769] obd_mod_free+0x46/0x140 [ptlrpc] [ 3756.075369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3756.075453] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3756.075460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3756.075463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3756.075477] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3756.075481] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3756.075486] RAX: 0000000000007d83 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3756.075489] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3756.075491] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3756.075493] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3756.075496] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3756.075509] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3756.075513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3756.075516] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3756.075523] Call Trace: [ 3756.075528] [ 3756.075533] ? show_regs.cold.9+0x22/0x2f [ 3756.075542] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3756.075548] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3756.075552] ? watchdog+0x40/0x40 [ 3756.075556] ? hrtimer_interrupt+0x128/0x2c0 [ 3756.075560] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3756.075568] ? apic_timer_interrupt+0xf/0x20 [ 3756.075573] [ 3756.075575] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3756.075582] do_raw_spin_lock+0x177/0x190 [ 3756.075586] _raw_spin_lock+0x21/0x30 [ 3756.075590] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3756.076129] ? do_raw_spin_unlock+0x75/0x190 [ 3756.076134] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3756.076333] ? do_wait_intr+0xf0/0xf0 [ 3756.076341] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3756.076554] mdc_close+0x53b/0xd70 [mdc] [ 3756.076592] lmv_close+0x18d/0x490 [lmv] [ 3756.076617] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3756.117402] mdc_commit_open+0x91/0xb0 [mdc] [ 3756.117450] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3756.117833] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3756.118032] after_reply+0xe2c/0x1780 [ptlrpc] [ 3756.118375] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3756.118656] ? __next_timer_interrupt+0x160/0x160 [ 3756.118666] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3756.118882] ? do_wait_intr+0xf0/0xf0 [ 3756.118892] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3756.133010] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3756.133253] ll_md_close+0x274/0x800 [lustre] [ 3756.133340] ll_file_release+0x128/0xc50 [lustre] [ 3756.133434] __fput+0x10d/0x440 [ 3756.133445] ____fput+0x12/0x20 [ 3756.133451] task_work_run+0xc6/0x110 [ 3756.133458] exit_to_usermode_loop+0x1dd/0x1f0 [ 3756.133466] do_syscall_64+0x3d6/0x440 [ 3756.133471] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3756.133480] RIP: 0033:0x7f1a63983cd8 [ 3756.133488] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3756.133492] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3756.133508] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3756.133512] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3756.133514] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3756.133516] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3756.133519] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3756.625159] kthread+0x1d1/0x200 [ 3756.626414] ? set_kthread_struct+0x70/0x70 [ 3756.631142] ret_from_fork+0x1f/0x30 [ 3760.053118] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3760.056047] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3760.060032] Modules linked in: [ 3760.063273] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3760.063350] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3760.063357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3760.063359] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3760.063373] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3760.063377] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3760.063381] RAX: 0000000000004984 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3760.063384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3760.063385] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3760.063388] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3760.063390] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3760.063393] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3760.063396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3760.063398] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3760.063406] Call Trace: [ 3760.063411] [ 3760.063417] ? show_regs.cold.9+0x22/0x2f [ 3760.063426] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3760.063432] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3760.063436] ? watchdog+0x40/0x40 [ 3760.063440] ? hrtimer_interrupt+0x128/0x2c0 [ 3760.063444] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3760.063452] ? apic_timer_interrupt+0xf/0x20 [ 3760.063523] [ 3760.063527] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3760.063533] do_raw_spin_lock+0x177/0x190 [ 3760.063538] _raw_spin_lock+0x21/0x30 [ 3760.063542] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3760.066991] lustre(O) [ 3760.083542] ? do_raw_spin_unlock+0x75/0x190 [ 3760.083564] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3760.100258] osp(O) [ 3760.104109] ? do_wait_intr+0xf0/0xf0 [ 3760.104130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3760.104394] mdc_close+0x53b/0xd70 [mdc] [ 3760.104576] lmv_close+0x18d/0x490 [lmv] [ 3760.104610] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3760.104722] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3760.104804] ll_md_close+0x274/0x800 [lustre] [ 3760.104891] ll_file_release+0x128/0xc50 [lustre] [ 3760.104974] __fput+0x10d/0x440 [ 3760.104983] ____fput+0x12/0x20 [ 3760.104987] task_work_run+0xc6/0x110 [ 3760.104995] exit_to_usermode_loop+0x1dd/0x1f0 [ 3760.105002] do_syscall_64+0x3d6/0x440 [ 3760.105007] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3760.105015] RIP: 0033:0x7f1a63983cd8 [ 3760.105024] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3760.105028] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3760.105034] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3760.105036] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3760.105038] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3760.105041] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3760.105043] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3760.243690] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3760.261816] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3760.267227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3760.270616] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3760.271741] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3760.277131] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3760.279457] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3760.282010] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3760.284625] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3760.286717] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3760.294476] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3760.297916] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3760.301206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3760.303911] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3760.316012] Call Trace: [ 3760.320814] [ 3760.324703] ? show_regs.cold.9+0x22/0x2f [ 3760.329562] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3760.331727] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3760.333953] ? watchdog+0x40/0x40 [ 3760.335320] ? hrtimer_interrupt+0x128/0x2c0 [ 3760.341805] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3760.347997] ? apic_timer_interrupt+0xf/0x20 [ 3760.352193] [ 3760.356313] ? native_safe_halt+0x2e/0x50 [ 3760.359200] kvm_wait+0xcd/0xe0 [ 3760.365426] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3760.367491] do_raw_spin_lock+0x177/0x190 [ 3760.368838] _raw_spin_lock+0x21/0x30 [ 3760.370771] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3760.377594] ? do_raw_spin_unlock+0x75/0x190 [ 3760.382999] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3760.388622] ? do_wait_intr+0xf0/0xf0 [ 3760.389955] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3760.394652] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3760.399980] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3760.406734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.414413] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3760.418404] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3760.420919] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3760.423270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.426895] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3760.431629] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3760.438325] lmv_intent_open+0x338/0x1150 [lmv] [ 3760.440318] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.443053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.445866] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3760.448315] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3760.450888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.454103] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3760.456107] ? do_raw_spin_unlock+0x75/0x190 [ 3760.458069] ? _raw_spin_unlock+0x12/0x30 [ 3760.461061] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3760.464858] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3760.467751] ll_atomic_open+0x36a/0xd70 [lustre] [ 3760.470333] lookup_open+0x356/0xb40 [ 3760.472208] do_last+0x313/0xfc0 [ 3760.473929] ? nd_jump_root+0xe5/0x160 [ 3760.475899] ? path_init+0x437/0x520 [ 3760.477628] path_openat+0xf7/0x500 [ 3760.480091] ? do_raw_spin_unlock+0x75/0x190 [ 3760.482953] do_filp_open+0x99/0x140 [ 3760.486223] ? getname_flags+0x6e/0x330 [ 3760.488738] ? __check_object_size+0xff/0x256 [ 3760.490777] ? do_raw_spin_unlock+0x75/0x190 [ 3760.493767] ? _raw_spin_unlock+0x12/0x30 [ 3760.496046] do_sys_openat2+0x2b4/0x410 [ 3760.498882] do_sys_open+0x73/0xa0 [ 3760.502869] __x64_sys_openat+0x24/0x30 [ 3760.507013] do_syscall_64+0xc1/0x440 [ 3760.509119] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3760.510772] RIP: 0033:0x7f1a63983332 [ 3760.523074] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3760.546608] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3760.553421] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3760.557918] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3760.567611] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3760.571383] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3760.577129] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3764.707906] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1741s! [ 3764.716628] Showing busy workqueues and worker pools: [ 3764.719581] workqueue events: flags=0x0 [ 3764.721927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3764.728611] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3764.737759] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3764.745877] pending: sec_gc_main [ptlrpc] [ 3764.749883] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3764.754931] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3764.762785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3764.773039] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3764.786392] workqueue events_long: flags=0x0 [ 3764.788687] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3764.793745] pending: lu_objects_flush [obdclass] [ 3764.796906] workqueue events_unbound: flags=0x2 [ 3764.799664] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3764.804223] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3764.811375] workqueue events_power_efficient: flags=0x80 [ 3764.815432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3764.822668] pending: neigh_periodic_work, do_cache_clean [ 3764.827498] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3764.830037] pending: neigh_periodic_work, check_lifetime [ 3764.844078] workqueue mm_percpu_wq: flags=0x8 [ 3764.846645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3764.853914] pending: vmstat_update [ 3764.856815] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3764.862071] pending: vmstat_update [ 3764.870877] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3764.874896] pending: vmstat_update [ 3764.877353] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3764.880948] pending: vmstat_update [ 3764.883729] workqueue writeback: flags=0x4a [ 3764.888251] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3764.894359] pending: wb_workfn, wb_workfn [ 3764.897567] workqueue xprtiod: flags=0xa [ 3764.900596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3764.907247] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3764.920768] workqueue ipv6_addrconf: flags=0x40008 [ 3764.922511] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3764.930535] pending: addrconf_verify_work [ 3764.932148] workqueue knbd0-recv: flags=0x1a [ 3764.937739] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3764.942019] in-flight: 647:recv_work [ 3764.945777] workqueue ptlrpc_pinger: flags=0x2 [ 3764.954949] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3764.957271] pending: ptlrpc_pinger_main [ptlrpc] [ 3764.959851] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3754s workers=2 idle: 652 [ 3784.057010] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3784.058032] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 3784.061508] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3784.061594] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3784.061601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3784.061604] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3784.061618] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3784.061622] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3784.061627] RAX: 0000000000003645 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3784.061630] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3784.061632] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3784.061634] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3784.061697] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3784.061701] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3784.061703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3784.061706] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3784.061714] Call Trace: [ 3784.061719] [ 3784.061724] ? show_regs.cold.9+0x22/0x2f [ 3784.061733] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3784.061739] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3784.061744] ? watchdog+0x40/0x40 [ 3784.061748] ? hrtimer_interrupt+0x128/0x2c0 [ 3784.061752] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3784.061760] ? apic_timer_interrupt+0xf/0x20 [ 3784.061765] [ 3784.061767] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3784.061774] do_raw_spin_lock+0x177/0x190 [ 3784.061778] _raw_spin_lock+0x21/0x30 [ 3784.061782] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3784.067244] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3784.067318] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3784.067325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3784.067328] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3784.067342] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3784.067346] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3784.067351] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3784.067353] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3784.067355] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3784.067357] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3784.067359] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3784.067362] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3784.067365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3784.067367] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3784.067375] Call Trace: [ 3784.067381] [ 3784.067386] ? show_regs.cold.9+0x22/0x2f [ 3784.067396] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3784.067402] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3784.067407] ? watchdog+0x40/0x40 [ 3784.067412] ? hrtimer_interrupt+0x128/0x2c0 [ 3784.067416] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3784.067422] ? apic_timer_interrupt+0xf/0x20 [ 3784.067427] [ 3784.067430] ? native_safe_halt+0x2e/0x50 [ 3784.067434] kvm_wait+0xcd/0xe0 [ 3784.067441] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3784.067448] do_raw_spin_lock+0x177/0x190 [ 3784.067453] _raw_spin_lock+0x21/0x30 [ 3784.067458] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3784.084814] ? do_raw_spin_unlock+0x75/0x190 [ 3784.084835] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3784.085121] ? do_wait_intr+0xf0/0xf0 [ 3784.085129] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3784.091428] obd_mod_free+0x46/0x140 [ptlrpc] [ 3784.096225] mdc_close+0x53b/0xd70 [mdc] [ 3784.101787] mdc_commit_open+0x91/0xb0 [mdc] [ 3784.101835] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3784.102083] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3784.102290] after_reply+0xe2c/0x1780 [ptlrpc] [ 3784.102439] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3784.102623] ? __next_timer_interrupt+0x160/0x160 [ 3784.102633] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3784.102833] ? do_wait_intr+0xf0/0xf0 [ 3784.102843] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3784.103041] kthread+0x1d1/0x200 [ 3784.103046] ? set_kthread_struct+0x70/0x70 [ 3784.103050] ret_from_fork+0x1f/0x30 [ 3784.450616] lmv_close+0x18d/0x490 [lmv] [ 3784.459759] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3784.466189] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3784.472574] ll_md_close+0x274/0x800 [lustre] [ 3784.476576] ll_file_release+0x128/0xc50 [lustre] [ 3784.481997] __fput+0x10d/0x440 [ 3784.485204] ____fput+0x12/0x20 [ 3784.491497] task_work_run+0xc6/0x110 [ 3784.501571] exit_to_usermode_loop+0x1dd/0x1f0 [ 3784.509933] do_syscall_64+0x3d6/0x440 [ 3784.511312] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3784.515572] RIP: 0033:0x7f1a63983cd8 [ 3784.518372] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3784.548496] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3784.557959] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3784.569192] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3784.575312] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3784.583747] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3784.593719] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3788.058593] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3788.067570] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3788.143028] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3788.152893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3788.167051] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3788.173468] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3788.206675] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3788.210700] RAX: 0000000000001015 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3788.223828] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3788.236128] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3788.250966] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3788.253448] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3788.270835] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3788.275749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3788.282452] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3788.291482] Call Trace: [ 3788.294585] [ 3788.296601] ? show_regs.cold.9+0x22/0x2f [ 3788.299042] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3788.303945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3788.309521] ? watchdog+0x40/0x40 [ 3788.310893] ? hrtimer_interrupt+0x128/0x2c0 [ 3788.315589] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3788.320724] ? apic_timer_interrupt+0xf/0x20 [ 3788.326106] [ 3788.328083] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3788.332913] do_raw_spin_lock+0x177/0x190 [ 3788.334789] _raw_spin_lock+0x21/0x30 [ 3788.337293] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3788.339129] ? do_raw_spin_unlock+0x75/0x190 [ 3788.341428] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3788.343097] ? do_wait_intr+0xf0/0xf0 [ 3788.348878] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3788.351908] mdc_close+0x53b/0xd70 [mdc] [ 3788.359875] lmv_close+0x18d/0x490 [lmv] [ 3788.361166] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3788.366062] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3788.368860] ll_md_close+0x274/0x800 [lustre] [ 3788.372980] ll_file_release+0x128/0xc50 [lustre] [ 3788.374593] __fput+0x10d/0x440 [ 3788.378269] ____fput+0x12/0x20 [ 3788.383817] task_work_run+0xc6/0x110 [ 3788.385924] exit_to_usermode_loop+0x1dd/0x1f0 [ 3788.388861] do_syscall_64+0x3d6/0x440 [ 3788.390699] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3788.392920] RIP: 0033:0x7f1a63983cd8 [ 3788.398803] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3788.417015] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3788.420875] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3788.425156] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3788.434960] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3788.439740] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3788.446438] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3792.053012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3792.055838] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3792.080369] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3792.084757] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3792.088897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3792.091754] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3792.099721] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3792.102869] RAX: 0000000000005f8a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3792.105990] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3792.110114] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3792.113718] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3792.117674] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3792.121865] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3792.126466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3792.129626] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3792.134559] Call Trace: [ 3792.136225] [ 3792.137560] ? show_regs.cold.9+0x22/0x2f [ 3792.139725] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3792.142468] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3792.145341] ? watchdog+0x40/0x40 [ 3792.148130] ? hrtimer_interrupt+0x128/0x2c0 [ 3792.149810] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3792.152697] ? apic_timer_interrupt+0xf/0x20 [ 3792.155337] [ 3792.156474] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3792.160196] do_raw_spin_lock+0x177/0x190 [ 3792.162258] _raw_spin_lock+0x21/0x30 [ 3792.164205] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3792.166795] ? do_raw_spin_unlock+0x75/0x190 [ 3792.169347] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3792.171298] ? do_wait_intr+0xf0/0xf0 [ 3792.173455] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3792.176268] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3792.179544] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3792.182571] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3792.185538] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3792.188255] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3792.190548] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3792.192747] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3792.196295] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3792.199233] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3792.201962] lmv_intent_open+0x338/0x1150 [lmv] [ 3792.204688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3792.210565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3792.213550] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3792.216210] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3792.218689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3792.222104] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3792.224952] ? do_raw_spin_unlock+0x75/0x190 [ 3792.227034] ? _raw_spin_unlock+0x12/0x30 [ 3792.228754] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3792.231609] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3792.233949] ll_atomic_open+0x36a/0xd70 [lustre] [ 3792.236327] lookup_open+0x356/0xb40 [ 3792.237696] do_last+0x313/0xfc0 [ 3792.239943] ? nd_jump_root+0xe5/0x160 [ 3792.242247] ? path_init+0x437/0x520 [ 3792.244282] path_openat+0xf7/0x500 [ 3792.246469] ? do_raw_spin_unlock+0x75/0x190 [ 3792.248652] do_filp_open+0x99/0x140 [ 3792.249972] ? getname_flags+0x6e/0x330 [ 3792.252771] ? __check_object_size+0xff/0x256 [ 3792.255333] ? do_raw_spin_unlock+0x75/0x190 [ 3792.257651] ? _raw_spin_unlock+0x12/0x30 [ 3792.260898] do_sys_openat2+0x2b4/0x410 [ 3792.263347] do_sys_open+0x73/0xa0 [ 3792.264722] __x64_sys_openat+0x24/0x30 [ 3792.270900] do_syscall_64+0xc1/0x440 [ 3792.273309] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3792.276817] RIP: 0033:0x7f1a63983332 [ 3792.278927] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3792.287298] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3792.290430] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3792.293801] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3792.297724] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3792.301618] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3792.305659] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3795.429606] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1772s! [ 3795.439257] Showing busy workqueues and worker pools: [ 3795.442028] workqueue events: flags=0x0 [ 3795.443736] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3795.447336] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3795.451258] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3795.454217] pending: sec_gc_main [ptlrpc] [ 3795.456402] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3795.459481] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3795.464054] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3795.467421] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3795.472733] workqueue events_long: flags=0x0 [ 3795.475223] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3795.478888] pending: lu_objects_flush [obdclass] [ 3795.481550] workqueue events_unbound: flags=0x2 [ 3795.483919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3795.486999] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3795.491695] workqueue events_power_efficient: flags=0x80 [ 3795.494171] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3795.497026] pending: neigh_periodic_work, do_cache_clean [ 3795.500017] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3795.502926] pending: neigh_periodic_work, check_lifetime [ 3795.505799] workqueue mm_percpu_wq: flags=0x8 [ 3795.507882] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3795.510925] pending: vmstat_update [ 3795.512740] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3795.516088] pending: vmstat_update [ 3795.518099] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3795.521180] pending: vmstat_update [ 3795.522960] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3795.526392] pending: vmstat_update [ 3795.528702] workqueue writeback: flags=0x4a [ 3795.530660] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3795.533618] pending: wb_workfn, wb_workfn [ 3795.535644] workqueue xprtiod: flags=0xa [ 3795.537767] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3795.540556] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3795.543785] workqueue ipv6_addrconf: flags=0x40008 [ 3795.546285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3795.549543] pending: addrconf_verify_work [ 3795.551615] workqueue knbd0-recv: flags=0x1a [ 3795.553549] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3795.556631] in-flight: 647:recv_work [ 3795.558553] workqueue ptlrpc_pinger: flags=0x2 [ 3795.560746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3795.563957] pending: ptlrpc_pinger_main [ptlrpc] [ 3795.566746] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3785s workers=2 idle: 652 [ 3812.059153] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 3812.060417] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3812.069602] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3812.069688] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3812.069698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3812.069701] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3812.069722] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3812.069727] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3812.069731] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3812.069734] RDX: 0000000000005034 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3812.069736] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3812.069739] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3812.069741] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3812.069744] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3812.069747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3812.069750] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3812.069758] Call Trace: [ 3812.069766] [ 3812.070193] ? show_regs.cold.9+0x22/0x2f [ 3812.070210] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3812.070218] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3812.070224] ? watchdog+0x40/0x40 [ 3812.070228] ? hrtimer_interrupt+0x128/0x2c0 [ 3812.070233] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3812.070244] ? apic_timer_interrupt+0xf/0x20 [ 3812.070248] [ 3812.070252] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3812.070257] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3812.070262] do_raw_spin_lock+0x177/0x190 [ 3812.070266] _raw_spin_lock+0x21/0x30 [ 3812.070288] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3812.070599] obd_mod_free+0x46/0x140 [ptlrpc] [ 3812.071938] mdc_commit_open+0x91/0xb0 [mdc] [ 3812.072599] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3812.073190] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3812.073847] after_reply+0xe2c/0x1780 [ptlrpc] [ 3812.074088] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3812.074298] ? __next_timer_interrupt+0x160/0x160 [ 3812.074310] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3812.074508] ? do_wait_intr+0xf0/0xf0 [ 3812.074520] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3812.079640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3812.079725] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3812.079733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3812.079736] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3812.079751] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3812.079755] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3812.079760] RAX: 0000000000007b34 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3812.079762] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3812.079764] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3812.079766] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3812.079769] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3812.079772] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3812.079775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3812.079778] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3812.079788] Call Trace: [ 3812.079793] [ 3812.079798] ? show_regs.cold.9+0x22/0x2f [ 3812.079810] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3812.079816] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3812.079822] ? watchdog+0x40/0x40 [ 3812.079826] ? hrtimer_interrupt+0x128/0x2c0 [ 3812.081933] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3812.129054] kthread+0x1d1/0x200 [ 3812.129609] ? set_kthread_struct+0x70/0x70 [ 3812.129614] ret_from_fork+0x1f/0x30 [ 3812.518659] ? apic_timer_interrupt+0xf/0x20 [ 3812.521064] [ 3812.522314] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3812.527186] do_raw_spin_lock+0x177/0x190 [ 3812.531817] _raw_spin_lock+0x21/0x30 [ 3812.535435] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3812.541318] ? do_raw_spin_unlock+0x75/0x190 [ 3812.546405] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3812.553713] ? do_wait_intr+0xf0/0xf0 [ 3812.560431] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3812.567412] mdc_close+0x53b/0xd70 [mdc] [ 3812.568858] lmv_close+0x18d/0x490 [lmv] [ 3812.573866] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3812.578441] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3812.584614] ll_md_close+0x274/0x800 [lustre] [ 3812.589265] ll_file_release+0x128/0xc50 [lustre] [ 3812.596238] __fput+0x10d/0x440 [ 3812.600471] ____fput+0x12/0x20 [ 3812.609431] task_work_run+0xc6/0x110 [ 3812.613329] exit_to_usermode_loop+0x1dd/0x1f0 [ 3812.617904] do_syscall_64+0x3d6/0x440 [ 3812.624308] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3812.627717] RIP: 0033:0x7f1a63983cd8 [ 3812.637650] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3812.662581] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3812.666334] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3812.685853] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3812.702066] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3812.715674] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3812.723828] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3816.056027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3816.059023] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3816.089200] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3816.094592] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3816.099282] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3816.101160] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3816.109170] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3816.114503] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3816.116870] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3816.122202] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3816.130075] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3816.134395] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3816.139875] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3816.147087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3816.152223] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3816.157663] Call Trace: [ 3816.159143] [ 3816.159582] ? show_regs.cold.9+0x22/0x2f [ 3816.161737] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3816.165085] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3816.168573] ? watchdog+0x40/0x40 [ 3816.171599] ? hrtimer_interrupt+0x128/0x2c0 [ 3816.175483] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3816.176693] ? apic_timer_interrupt+0xf/0x20 [ 3816.177695] [ 3816.180764] ? native_safe_halt+0x2e/0x50 [ 3816.182834] kvm_wait+0xcd/0xe0 [ 3816.184399] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3816.187447] do_raw_spin_lock+0x177/0x190 [ 3816.189402] _raw_spin_lock+0x21/0x30 [ 3816.195617] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3816.204732] ? do_raw_spin_unlock+0x75/0x190 [ 3816.207758] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3816.211847] ? do_wait_intr+0xf0/0xf0 [ 3816.216705] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3816.220713] mdc_close+0x53b/0xd70 [mdc] [ 3816.223373] lmv_close+0x18d/0x490 [lmv] [ 3816.226716] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3816.228898] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3816.233550] ll_md_close+0x274/0x800 [lustre] [ 3816.236255] ll_file_release+0x128/0xc50 [lustre] [ 3816.238119] __fput+0x10d/0x440 [ 3816.240585] ____fput+0x12/0x20 [ 3816.245591] task_work_run+0xc6/0x110 [ 3816.248216] exit_to_usermode_loop+0x1dd/0x1f0 [ 3816.250293] do_syscall_64+0x3d6/0x440 [ 3816.252781] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3816.255746] RIP: 0033:0x7f1a63983cd8 [ 3816.259545] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3816.270310] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3816.272280] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3816.273909] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3816.281310] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3816.290053] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3816.292755] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3820.053055] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3820.061574] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3820.095534] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3820.100537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3820.105510] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [ 3820.112262] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [ 3820.122562] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3820.126260] RAX: 0000000000000e2d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3820.129731] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3820.133302] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3820.136345] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3820.139648] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3820.142897] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3820.146721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3820.149752] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3820.154713] Call Trace: [ 3820.156498] [ 3820.157311] ? show_regs.cold.9+0x22/0x2f [ 3820.159280] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3820.161094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3820.164130] ? watchdog+0x40/0x40 [ 3820.165656] ? hrtimer_interrupt+0x128/0x2c0 [ 3820.168320] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3820.171261] ? apic_timer_interrupt+0xf/0x20 [ 3820.174589] [ 3820.176415] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [ 3820.179632] do_raw_spin_lock+0x177/0x190 [ 3820.181730] _raw_spin_lock+0x21/0x30 [ 3820.184295] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3820.186798] ? do_raw_spin_unlock+0x75/0x190 [ 3820.190009] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3820.192282] ? do_wait_intr+0xf0/0xf0 [ 3820.193615] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3820.195837] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3820.198113] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3820.201381] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3820.203650] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3820.206086] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3820.208087] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3820.209835] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3820.212475] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3820.216271] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3820.219117] lmv_intent_open+0x338/0x1150 [lmv] [ 3820.221112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3820.223528] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3820.225930] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3820.227803] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3820.230074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3820.232364] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3820.235101] ? do_raw_spin_unlock+0x75/0x190 [ 3820.237176] ? _raw_spin_unlock+0x12/0x30 [ 3820.238845] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3820.241256] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3820.244140] ll_atomic_open+0x36a/0xd70 [lustre] [ 3820.246751] lookup_open+0x356/0xb40 [ 3820.248593] do_last+0x313/0xfc0 [ 3820.250230] ? nd_jump_root+0xe5/0x160 [ 3820.251818] ? path_init+0x437/0x520 [ 3820.254315] path_openat+0xf7/0x500 [ 3820.255715] ? do_raw_spin_unlock+0x75/0x190 [ 3820.257659] do_filp_open+0x99/0x140 [ 3820.259333] ? getname_flags+0x6e/0x330 [ 3820.261247] ? __check_object_size+0xff/0x256 [ 3820.263183] ? do_raw_spin_unlock+0x75/0x190 [ 3820.265033] ? _raw_spin_unlock+0x12/0x30 [ 3820.266798] do_sys_openat2+0x2b4/0x410 [ 3820.268454] do_sys_open+0x73/0xa0 [ 3820.270737] __x64_sys_openat+0x24/0x30 [ 3820.272314] do_syscall_64+0xc1/0x440 [ 3820.274247] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3820.276556] RIP: 0033:0x7f1a63983332 [ 3820.278175] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3820.286571] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3820.289671] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3820.292244] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3820.295745] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3820.298053] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3820.300676] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3825.633065] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1802s! [ 3825.636773] Showing busy workqueues and worker pools: [ 3825.638911] workqueue events: flags=0x0 [ 3825.640611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3825.643785] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3825.647279] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3825.650519] pending: sec_gc_main [ptlrpc] [ 3825.652886] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3825.656031] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3825.661723] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3825.665478] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3825.669959] workqueue events_long: flags=0x0 [ 3825.672662] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3825.676084] pending: lu_objects_flush [obdclass] [ 3825.678698] workqueue events_unbound: flags=0x2 [ 3825.681010] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3825.683585] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3825.688785] workqueue events_power_efficient: flags=0x80 [ 3825.691601] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3825.695097] pending: neigh_periodic_work, do_cache_clean [ 3825.698186] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3825.701600] pending: neigh_periodic_work, check_lifetime [ 3825.704326] workqueue mm_percpu_wq: flags=0x8 [ 3825.706285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3825.709215] pending: vmstat_update [ 3825.711359] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3825.714413] pending: vmstat_update [ 3825.716195] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3825.719413] pending: vmstat_update [ 3825.721251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3825.724614] pending: vmstat_update [ 3825.726589] workqueue writeback: flags=0x4a [ 3825.729449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3825.733296] pending: wb_workfn, wb_workfn [ 3825.735441] workqueue xprtiod: flags=0xa [ 3825.737499] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3825.740554] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3825.744799] workqueue ipv6_addrconf: flags=0x40008 [ 3825.747343] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3825.750516] pending: addrconf_verify_work [ 3825.753329] workqueue knbd0-recv: flags=0x1a [ 3825.756128] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3825.759332] in-flight: 647:recv_work [ 3825.761558] workqueue ptlrpc_pinger: flags=0x2 [ 3825.764066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3825.767442] pending: ptlrpc_pinger_main [ptlrpc] [ 3825.770029] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3815s workers=2 idle: 652 [ 3840.057185] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 3840.058012] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 3840.062663] Modules linked in: lustre(O) [ 3840.066168] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [ 3840.070150] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 3840.090174] sb_edac [ 3840.090176] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 3840.103319] rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3840.103365] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3840.103372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3840.103374] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3840.103409] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3840.103413] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3840.103417] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3840.103419] RDX: 000000000000570f RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3840.103421] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3840.103423] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3840.103425] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3840.103428] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3840.103431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3840.103433] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3840.103440] Call Trace: [ 3840.103451] [ 3840.103456] ? show_regs.cold.9+0x22/0x2f [ 3840.103465] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3840.103471] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3840.103474] ? watchdog+0x40/0x40 [ 3840.103478] ? hrtimer_interrupt+0x128/0x2c0 [ 3840.103481] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3840.103488] ? apic_timer_interrupt+0xf/0x20 [ 3840.103492] [ 3840.103494] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3840.103498] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3840.103502] do_raw_spin_lock+0x177/0x190 [ 3840.103506] _raw_spin_lock+0x21/0x30 [ 3840.103510] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3840.104557] intel_rapl_common [ 3840.107117] obd_mod_free+0x46/0x140 [ptlrpc] [ 3840.120443] sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3840.120478] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3840.120484] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3840.120487] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3840.120499] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3840.120504] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3840.120508] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3840.120510] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3840.120512] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3840.120514] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3840.120516] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3840.120520] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3840.120522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3840.120525] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3840.120532] Call Trace: [ 3840.120537] [ 3840.120541] ? show_regs.cold.9+0x22/0x2f [ 3840.120549] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3840.120555] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3840.120559] ? watchdog+0x40/0x40 [ 3840.120563] ? hrtimer_interrupt+0x128/0x2c0 [ 3840.120567] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3840.120575] ? apic_timer_interrupt+0xf/0x20 [ 3840.125605] mdc_commit_open+0x91/0xb0 [mdc] [ 3840.128921] [ 3840.131492] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3840.143006] ? native_safe_halt+0x2e/0x50 [ 3840.146350] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3840.149029] kvm_wait+0xcd/0xe0 [ 3840.152285] after_reply+0xe2c/0x1780 [ptlrpc] [ 3840.152630] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3840.152909] ? __next_timer_interrupt+0x160/0x160 [ 3840.152920] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3840.153128] ? do_wait_intr+0xf0/0xf0 [ 3840.153137] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3840.153403] kthread+0x1d1/0x200 [ 3840.153409] ? set_kthread_struct+0x70/0x70 [ 3840.153413] ret_from_fork+0x1f/0x30 [ 3840.304260] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3840.306094] do_raw_spin_lock+0x177/0x190 [ 3840.307443] _raw_spin_lock+0x21/0x30 [ 3840.308761] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3840.310781] ? do_raw_spin_unlock+0x75/0x190 [ 3840.312180] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3840.314254] ? do_wait_intr+0xf0/0xf0 [ 3840.315509] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3840.317607] mdc_close+0x53b/0xd70 [mdc] [ 3840.318881] lmv_close+0x18d/0x490 [lmv] [ 3840.320405] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3840.323477] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3840.326031] ll_md_close+0x274/0x800 [lustre] [ 3840.328567] ll_file_release+0x128/0xc50 [lustre] [ 3840.330825] __fput+0x10d/0x440 [ 3840.332361] ____fput+0x12/0x20 [ 3840.333549] task_work_run+0xc6/0x110 [ 3840.335118] exit_to_usermode_loop+0x1dd/0x1f0 [ 3840.337020] do_syscall_64+0x3d6/0x440 [ 3840.338747] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3840.341478] RIP: 0033:0x7f1a63983cd8 [ 3840.342829] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3840.363236] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3840.367718] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3840.371583] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3840.375438] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3840.379194] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3840.382422] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3844.060371] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 3844.068776] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3844.125499] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3844.131299] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3844.135111] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3844.137584] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 3844.150194] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3844.160155] RAX: 00000000000054b1 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3844.162639] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3844.169970] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3844.178423] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3844.182996] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3844.189806] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3844.193856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3844.198261] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3844.203009] Call Trace: [ 3844.204570] [ 3844.205342] ? show_regs.cold.9+0x22/0x2f [ 3844.208082] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3844.213274] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3844.219133] ? watchdog+0x40/0x40 [ 3844.222241] ? hrtimer_interrupt+0x128/0x2c0 [ 3844.226345] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3844.229116] ? apic_timer_interrupt+0xf/0x20 [ 3844.231922] [ 3844.232886] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3844.235689] do_raw_spin_lock+0x177/0x190 [ 3844.237313] _raw_spin_lock+0x21/0x30 [ 3844.239109] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3844.244084] ? do_raw_spin_unlock+0x75/0x190 [ 3844.245536] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3844.248733] ? do_wait_intr+0xf0/0xf0 [ 3844.250210] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3844.253266] mdc_close+0x53b/0xd70 [mdc] [ 3844.256037] lmv_close+0x18d/0x490 [lmv] [ 3844.257582] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3844.262027] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3844.266718] ll_md_close+0x274/0x800 [lustre] [ 3844.269498] ll_file_release+0x128/0xc50 [lustre] [ 3844.272842] __fput+0x10d/0x440 [ 3844.274876] ____fput+0x12/0x20 [ 3844.276944] task_work_run+0xc6/0x110 [ 3844.279858] exit_to_usermode_loop+0x1dd/0x1f0 [ 3844.281358] do_syscall_64+0x3d6/0x440 [ 3844.282801] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3844.284914] RIP: 0033:0x7f1a63983cd8 [ 3844.287261] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3844.295498] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3844.301099] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3844.304878] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3844.310460] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3844.313514] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3844.317602] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3852.060660] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3852.065197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3852.107997] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3852.124122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3852.131207] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3852.139508] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3852.160081] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3852.163484] RAX: 000000000000080c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3852.168968] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3852.173877] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3852.179480] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3852.184853] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3852.187248] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3852.191115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3852.194536] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3852.198348] Call Trace: [ 3852.199699] [ 3852.201177] ? show_regs.cold.9+0x22/0x2f [ 3852.202474] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3852.204235] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3852.206310] ? watchdog+0x40/0x40 [ 3852.207733] ? hrtimer_interrupt+0x128/0x2c0 [ 3852.209408] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3852.211410] ? apic_timer_interrupt+0xf/0x20 [ 3852.213363] [ 3852.214289] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3852.216724] do_raw_spin_lock+0x177/0x190 [ 3852.218327] _raw_spin_lock+0x21/0x30 [ 3852.219936] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3852.222392] ? do_raw_spin_unlock+0x75/0x190 [ 3852.224108] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3852.226389] ? do_wait_intr+0xf0/0xf0 [ 3852.227956] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3852.230269] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3852.235976] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3852.241048] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3852.246290] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3852.249535] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3852.254953] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3852.257301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3852.260534] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3852.268006] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3852.272952] lmv_intent_open+0x338/0x1150 [lmv] [ 3852.276117] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3852.281723] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3852.285063] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3852.287145] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3852.289652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3852.293237] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3852.296006] ? do_raw_spin_unlock+0x75/0x190 [ 3852.299115] ? _raw_spin_unlock+0x12/0x30 [ 3852.301473] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3852.304588] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3852.311163] ll_atomic_open+0x36a/0xd70 [lustre] [ 3852.314135] lookup_open+0x356/0xb40 [ 3852.316712] do_last+0x313/0xfc0 [ 3852.317972] ? nd_jump_root+0xe5/0x160 [ 3852.320733] ? path_init+0x437/0x520 [ 3852.322825] path_openat+0xf7/0x500 [ 3852.324757] ? do_raw_spin_unlock+0x75/0x190 [ 3852.327057] do_filp_open+0x99/0x140 [ 3852.329149] ? getname_flags+0x6e/0x330 [ 3852.331127] ? __check_object_size+0xff/0x256 [ 3852.333841] ? do_raw_spin_unlock+0x75/0x190 [ 3852.336062] ? _raw_spin_unlock+0x12/0x30 [ 3852.338742] do_sys_openat2+0x2b4/0x410 [ 3852.341404] do_sys_open+0x73/0xa0 [ 3852.342887] __x64_sys_openat+0x24/0x30 [ 3852.344245] do_syscall_64+0xc1/0x440 [ 3852.346517] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3852.349060] RIP: 0033:0x7f1a63983332 [ 3852.350674] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3852.361388] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3852.365144] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3852.369027] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3852.377002] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3852.383822] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3852.387833] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3855.841639] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1832s! [ 3855.847690] Showing busy workqueues and worker pools: [ 3855.858433] workqueue events: flags=0x0 [ 3855.859730] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3855.868431] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3855.886786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3855.896461] pending: sec_gc_main [ptlrpc] [ 3855.904213] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3855.912828] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3855.925463] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3855.935940] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3855.948610] workqueue events_long: flags=0x0 [ 3855.950211] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3855.962085] pending: lu_objects_flush [obdclass] [ 3855.966843] workqueue events_unbound: flags=0x2 [ 3855.972426] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3855.987116] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3855.999629] workqueue events_power_efficient: flags=0x80 [ 3856.004635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3856.008664] pending: neigh_periodic_work, do_cache_clean [ 3856.013042] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3856.016357] pending: neigh_periodic_work, check_lifetime [ 3856.021784] workqueue mm_percpu_wq: flags=0x8 [ 3856.024684] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3856.036083] pending: vmstat_update [ 3856.037596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3856.050665] pending: vmstat_update [ 3856.054748] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3856.062236] pending: vmstat_update [ 3856.067427] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3856.075417] pending: vmstat_update [ 3856.076881] workqueue writeback: flags=0x4a [ 3856.080182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3856.091125] pending: wb_workfn, wb_workfn [ 3856.093775] workqueue xprtiod: flags=0xa [ 3856.097895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3856.101340] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3856.107925] workqueue ipv6_addrconf: flags=0x40008 [ 3856.110628] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3856.114423] pending: addrconf_verify_work [ 3856.119266] workqueue knbd0-recv: flags=0x1a [ 3856.123496] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3856.129592] in-flight: 647:recv_work [ 3856.132111] workqueue ptlrpc_pinger: flags=0x2 [ 3856.136374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3856.141355] pending: ptlrpc_pinger_main [ptlrpc] [ 3856.145927] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3846s workers=2 idle: 652 [ 3868.057021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3868.058064] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3868.058958] Modules linked in: [ 3868.061213] Modules linked in: [ 3868.061216] lustre(O) osp(O) ofd(O) [ 3868.062121] lustre(O) [ 3868.063155] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3868.063231] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3868.063239] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3868.063241] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3868.063255] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3868.063259] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3868.063267] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3868.063269] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3868.063271] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3868.063274] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3868.063276] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3868.063279] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3868.063282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3868.063285] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3868.063293] Call Trace: [ 3868.063298] [ 3868.063303] ? show_regs.cold.9+0x22/0x2f [ 3868.063314] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3868.063320] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3868.063325] ? watchdog+0x40/0x40 [ 3868.063329] ? hrtimer_interrupt+0x128/0x2c0 [ 3868.063334] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3868.063341] ? apic_timer_interrupt+0xf/0x20 [ 3868.063345] [ 3868.063347] ? native_safe_halt+0x2e/0x50 [ 3868.063351] kvm_wait+0xcd/0xe0 [ 3868.063359] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3868.063367] do_raw_spin_lock+0x177/0x190 [ 3868.063372] _raw_spin_lock+0x21/0x30 [ 3868.063377] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3868.063815] ? do_raw_spin_unlock+0x75/0x190 [ 3868.063821] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3868.066100] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3868.066181] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3868.066189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3868.066192] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3868.066204] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3868.066208] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3868.066213] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3868.066215] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3868.066217] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3868.066219] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3868.066222] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3868.066225] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3868.066228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3868.066230] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3868.066241] Call Trace: [ 3868.066246] [ 3868.066251] ? show_regs.cold.9+0x22/0x2f [ 3868.066266] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3868.066272] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3868.066276] ? watchdog+0x40/0x40 [ 3868.066280] ? hrtimer_interrupt+0x128/0x2c0 [ 3868.066285] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3868.066291] ? apic_timer_interrupt+0xf/0x20 [ 3868.066295] [ 3868.066298] ? native_safe_halt+0x2e/0x50 [ 3868.066302] kvm_wait+0xcd/0xe0 [ 3868.066309] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3868.066316] do_raw_spin_lock+0x177/0x190 [ 3868.066320] _raw_spin_lock+0x21/0x30 [ 3868.066324] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3868.096577] ? do_wait_intr+0xf0/0xf0 [ 3868.096599] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3868.096908] mdc_close+0x53b/0xd70 [mdc] [ 3868.097065] lmv_close+0x18d/0x490 [lmv] [ 3868.097102] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3868.097228] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3868.097318] ll_md_close+0x274/0x800 [lustre] [ 3868.097399] ll_file_release+0x128/0xc50 [lustre] [ 3868.097546] __fput+0x10d/0x440 [ 3868.097606] ____fput+0x12/0x20 [ 3868.097610] task_work_run+0xc6/0x110 [ 3868.097619] exit_to_usermode_loop+0x1dd/0x1f0 [ 3868.097626] do_syscall_64+0x3d6/0x440 [ 3868.097647] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3868.097657] RIP: 0033:0x7f1a63983cd8 [ 3868.097664] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3868.097669] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3868.097678] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3868.097680] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3868.097683] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3868.097685] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3868.097687] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3868.340082] obd_mod_free+0x46/0x140 [ptlrpc] [ 3868.342081] mdc_commit_open+0x91/0xb0 [mdc] [ 3868.343727] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3868.346016] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3868.348110] after_reply+0xe2c/0x1780 [ptlrpc] [ 3868.349884] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3868.352326] ? __next_timer_interrupt+0x160/0x160 [ 3868.354144] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3868.356292] ? do_wait_intr+0xf0/0xf0 [ 3868.357882] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3868.360492] kthread+0x1d1/0x200 [ 3868.361763] ? set_kthread_struct+0x70/0x70 [ 3868.363736] ret_from_fork+0x1f/0x30 [ 3872.056092] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 3872.060002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3872.081955] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3872.088054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3872.092868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3872.097904] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3872.111863] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3872.119011] RAX: 000000000000229d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3872.123101] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3872.128031] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3872.134068] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3872.140147] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3872.143877] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3872.147576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3872.150496] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3872.154470] Call Trace: [ 3872.155839] [ 3872.157094] ? show_regs.cold.9+0x22/0x2f [ 3872.159255] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3872.161269] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3872.163635] ? watchdog+0x40/0x40 [ 3872.165313] ? hrtimer_interrupt+0x128/0x2c0 [ 3872.167353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3872.169266] ? apic_timer_interrupt+0xf/0x20 [ 3872.170814] [ 3872.171409] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3872.174048] do_raw_spin_lock+0x177/0x190 [ 3872.175242] _raw_spin_lock+0x21/0x30 [ 3872.176975] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3872.179319] ? do_raw_spin_unlock+0x75/0x190 [ 3872.180995] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3872.183067] ? do_wait_intr+0xf0/0xf0 [ 3872.184250] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3872.186577] mdc_close+0x53b/0xd70 [mdc] [ 3872.188932] lmv_close+0x18d/0x490 [lmv] [ 3872.190803] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3872.193064] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3872.194996] ll_md_close+0x274/0x800 [lustre] [ 3872.196563] ll_file_release+0x128/0xc50 [lustre] [ 3872.198907] __fput+0x10d/0x440 [ 3872.200116] ____fput+0x12/0x20 [ 3872.201120] task_work_run+0xc6/0x110 [ 3872.202307] exit_to_usermode_loop+0x1dd/0x1f0 [ 3872.203816] do_syscall_64+0x3d6/0x440 [ 3872.204993] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3872.206548] RIP: 0033:0x7f1a63983cd8 [ 3872.207713] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3872.215321] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3872.218944] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3872.222029] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3872.225831] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3872.229069] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3872.232168] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3882.450646] rcu: INFO: rcu_sched self-detected stall on CPU [ 3882.454723] rcu: 3-...!: (1106138 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 3882.462688] (t=1860052 jiffies g=269633 q=785) [ 3882.465363] rcu: rcu_sched kthread starved for 1859698 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 3882.474973] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3882.480651] rcu: RCU grace-period kthread stack dump: [ 3882.483879] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3882.492663] Call Trace: [ 3882.497717] __schedule+0x351/0xcb0 [ 3882.500892] schedule+0xc0/0x180 [ 3882.504958] schedule_timeout+0xb4/0x190 [ 3882.509678] ? do_raw_spin_unlock+0x75/0x190 [ 3882.511487] ? __next_timer_interrupt+0x160/0x160 [ 3882.514700] ? prepare_to_swait_event+0xf3/0x140 [ 3882.517523] rcu_gp_kthread+0x8dd/0x15b0 [ 3882.519755] ? force_qs_rnp+0x2c0/0x2c0 [ 3882.522211] kthread+0x1d1/0x200 [ 3882.523496] ? set_kthread_struct+0x70/0x70 [ 3882.525533] ret_from_fork+0x1f/0x30 [ 3882.527473] rcu: Stack dump where RCU GP kthread last ran: [ 3882.530190] Sending NMI from CPU 3 to CPUs 1: [ 3882.538341] NMI backtrace for cpu 1 [ 3882.538347] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3882.538350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3882.538351] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3882.538353] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3882.538355] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 [ 3882.538358] RAX: 0000000000007b86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3882.538359] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3882.538361] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3882.538362] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3882.538363] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3882.538365] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3882.538366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3882.538367] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3882.538368] Call Trace: [ 3882.538369] [ 3882.538370] ? show_regs.cold.9+0x22/0x2f [ 3882.538371] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3882.538371] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3882.538373] ? nmi_handle+0xbd/0x210 [ 3882.538373] ? default_do_nmi+0x4b/0x1b0 [ 3882.538374] ? do_nmi+0x280/0x380 [ 3882.538606] ? end_repeat_nmi+0x16/0x1f [ 3882.538607] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3882.538608] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3882.538609] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3882.538610] [ 3882.538611] do_raw_spin_lock+0x177/0x190 [ 3882.538612] _raw_spin_lock+0x21/0x30 [ 3882.538613] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3882.538613] ? do_raw_spin_unlock+0x75/0x190 [ 3882.538614] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3882.538615] ? do_wait_intr+0xf0/0xf0 [ 3882.538616] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3882.538617] mdc_close+0x53b/0xd70 [mdc] [ 3882.538618] lmv_close+0x18d/0x490 [lmv] [ 3882.538618] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3882.538619] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3882.538620] ll_md_close+0x274/0x800 [lustre] [ 3882.538621] ll_file_release+0x128/0xc50 [lustre] [ 3882.538622] __fput+0x10d/0x440 [ 3882.538622] ____fput+0x12/0x20 [ 3882.538623] task_work_run+0xc6/0x110 [ 3882.538624] exit_to_usermode_loop+0x1dd/0x1f0 [ 3882.538625] do_syscall_64+0x3d6/0x440 [ 3882.538626] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3882.538627] RIP: 0033:0x7f1a63983cd8 [ 3882.538628] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3882.538629] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3882.538632] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3882.538633] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3882.538634] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3882.538635] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3882.538636] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3882.538641] NMI backtrace for cpu 3 [ 3882.794647] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3882.802073] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3882.806569] Call Trace: [ 3882.807530] [ 3882.812311] ? dump_stack+0xbb/0x10e [ 3882.813535] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3882.816377] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3882.823503] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3882.828832] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3882.832299] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3882.841569] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3882.845970] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3882.849907] ? update_process_times+0x6d/0xc0 [ 3882.852777] ? tick_sched_handle.isra.18+0x27/0x90 [ 3882.858678] ? tick_sched_timer+0x53/0xb0 [ 3882.860808] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3882.863205] ? tick_sched_handle.isra.18+0x90/0x90 [ 3882.865683] ? hrtimer_interrupt+0x128/0x2c0 [ 3882.867715] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3882.869741] ? apic_timer_interrupt+0xf/0x20 [ 3882.871497] [ 3882.872298] ? __pv_queued_spin_lock_slowpath+0x17c/0x6d0 [ 3882.874774] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3882.876886] ? do_raw_spin_lock+0x177/0x190 [ 3882.878801] ? _raw_spin_lock+0x21/0x30 [ 3882.880436] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3882.881954] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3882.883712] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3882.885708] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3882.888793] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3882.891759] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3882.904152] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3882.909718] ? __next_timer_interrupt+0x160/0x160 [ 3882.914745] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3882.919715] ? do_wait_intr+0xf0/0xf0 [ 3882.921682] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3882.925505] ? kthread+0x1d1/0x200 [ 3882.926833] ? set_kthread_struct+0x70/0x70 [ 3882.931303] ? ret_from_fork+0x1f/0x30 [ 3884.055937] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3884.065365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3884.112963] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3884.124062] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3884.133132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3884.137942] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3884.149898] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3884.168229] RAX: 000000000000256c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3884.175982] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3884.185772] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3884.189892] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3884.192873] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3884.195830] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3884.200098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3884.202135] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3884.204789] Call Trace: [ 3884.205797] [ 3884.206404] ? show_regs.cold.9+0x22/0x2f [ 3884.207730] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3884.209430] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3884.210914] ? watchdog+0x40/0x40 [ 3884.212345] ? hrtimer_interrupt+0x128/0x2c0 [ 3884.213699] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3884.215417] ? apic_timer_interrupt+0xf/0x20 [ 3884.216748] [ 3884.217303] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3884.219050] do_raw_spin_lock+0x177/0x190 [ 3884.220331] _raw_spin_lock+0x21/0x30 [ 3884.221659] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3884.223826] ? do_raw_spin_unlock+0x75/0x190 [ 3884.224971] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3884.227078] ? do_wait_intr+0xf0/0xf0 [ 3884.228203] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3884.231198] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3884.232993] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3884.235171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3884.237060] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3884.238608] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3884.239914] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3884.241588] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3884.243093] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3884.244938] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3884.246647] lmv_intent_open+0x338/0x1150 [lmv] [ 3884.248368] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3884.249998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3884.251657] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3884.253104] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3884.260679] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3884.264204] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3884.267820] ? do_raw_spin_unlock+0x75/0x190 [ 3884.271461] ? _raw_spin_unlock+0x12/0x30 [ 3884.273704] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3884.278752] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3884.281997] ll_atomic_open+0x36a/0xd70 [lustre] [ 3884.287772] lookup_open+0x356/0xb40 [ 3884.290071] do_last+0x313/0xfc0 [ 3884.291818] ? nd_jump_root+0xe5/0x160 [ 3884.293039] ? path_init+0x437/0x520 [ 3884.294343] path_openat+0xf7/0x500 [ 3884.303791] ? do_raw_spin_unlock+0x75/0x190 [ 3884.305000] do_filp_open+0x99/0x140 [ 3884.305982] ? getname_flags+0x6e/0x330 [ 3884.307039] ? __check_object_size+0xff/0x256 [ 3884.313444] ? do_raw_spin_unlock+0x75/0x190 [ 3884.316454] ? _raw_spin_unlock+0x12/0x30 [ 3884.319274] do_sys_openat2+0x2b4/0x410 [ 3884.320602] do_sys_open+0x73/0xa0 [ 3884.321757] __x64_sys_openat+0x24/0x30 [ 3884.327380] do_syscall_64+0xc1/0x440 [ 3884.329651] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3884.331817] RIP: 0033:0x7f1a63983332 [ 3884.333797] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3884.353517] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3884.359489] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3884.370607] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3884.376655] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3884.380921] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3884.386886] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3886.560329] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1863s! [ 3886.564707] Showing busy workqueues and worker pools: [ 3886.567574] workqueue events: flags=0x0 [ 3886.569704] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3886.572911] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3886.575676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.578888] pending: sec_gc_main [ptlrpc] [ 3886.581703] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3886.584898] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3886.589888] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3886.597655] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3886.604715] workqueue events_long: flags=0x0 [ 3886.606814] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.610365] pending: lu_objects_flush [obdclass] [ 3886.612823] workqueue events_unbound: flags=0x2 [ 3886.615374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3886.618801] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3886.624355] workqueue events_power_efficient: flags=0x80 [ 3886.627209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3886.631889] pending: neigh_periodic_work, do_cache_clean [ 3886.635214] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3886.638840] pending: neigh_periodic_work, check_lifetime [ 3886.641617] workqueue mm_percpu_wq: flags=0x8 [ 3886.643667] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.647610] pending: vmstat_update [ 3886.649460] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.652413] pending: vmstat_update [ 3886.653903] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.657456] pending: vmstat_update [ 3886.659448] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.662695] pending: vmstat_update [ 3886.664500] workqueue writeback: flags=0x4a [ 3886.667437] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3886.670709] pending: wb_workfn, wb_workfn [ 3886.676207] workqueue xprtiod: flags=0xa [ 3886.681995] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3886.688916] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3886.700674] workqueue ipv6_addrconf: flags=0x40008 [ 3886.702859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3886.706385] pending: addrconf_verify_work [ 3886.708684] workqueue knbd0-recv: flags=0x1a [ 3886.710720] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3886.713817] in-flight: 647:recv_work [ 3886.715179] workqueue ptlrpc_pinger: flags=0x2 [ 3886.717896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3886.721488] pending: ptlrpc_pinger_main [ptlrpc] [ 3886.724979] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3876s workers=2 idle: 652 [ 3896.058236] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3896.065904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3896.089761] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3896.094859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3896.098376] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3896.101125] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3896.109709] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3896.113243] RAX: 0000000000006d87 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3896.116484] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3896.119849] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3896.122982] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3896.126310] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3896.129368] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3896.133011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3896.137879] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3896.145975] Call Trace: [ 3896.150346] [ 3896.151655] ? show_regs.cold.9+0x22/0x2f [ 3896.154303] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3896.157283] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3896.159971] ? watchdog+0x40/0x40 [ 3896.161355] ? hrtimer_interrupt+0x128/0x2c0 [ 3896.164375] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3896.167933] ? apic_timer_interrupt+0xf/0x20 [ 3896.169691] [ 3896.170754] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3896.174075] do_raw_spin_lock+0x177/0x190 [ 3896.175970] _raw_spin_lock+0x21/0x30 [ 3896.178637] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3896.182520] ? do_raw_spin_unlock+0x75/0x190 [ 3896.184880] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3896.187150] ? do_wait_intr+0xf0/0xf0 [ 3896.189504] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3896.193988] mdc_close+0x53b/0xd70 [mdc] [ 3896.198662] lmv_close+0x18d/0x490 [lmv] [ 3896.200412] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3896.202696] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3896.204853] ll_md_close+0x274/0x800 [lustre] [ 3896.207925] ll_file_release+0x128/0xc50 [lustre] [ 3896.210539] __fput+0x10d/0x440 [ 3896.212038] ____fput+0x12/0x20 [ 3896.213847] task_work_run+0xc6/0x110 [ 3896.215627] exit_to_usermode_loop+0x1dd/0x1f0 [ 3896.218300] do_syscall_64+0x3d6/0x440 [ 3896.220365] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3896.222848] RIP: 0033:0x7f1a63983cd8 [ 3896.224445] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3896.232918] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3896.235976] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3896.238986] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3896.242510] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3896.245860] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3896.249422] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3908.056458] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3908.058023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3908.061826] Modules linked in: lustre(O) [ 3908.070518] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3908.070596] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3908.070604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3908.070607] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3908.070621] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3908.070625] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3908.070629] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3908.070631] RDX: 00000000000036ab RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3908.070633] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3908.070635] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3908.070637] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3908.070641] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3908.070643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3908.070646] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3908.070655] Call Trace: [ 3908.070660] [ 3908.070666] ? show_regs.cold.9+0x22/0x2f [ 3908.070676] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3908.070683] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3908.070687] ? watchdog+0x40/0x40 [ 3908.070692] ? hrtimer_interrupt+0x128/0x2c0 [ 3908.070697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3908.070705] ? apic_timer_interrupt+0xf/0x20 [ 3908.070710] [ 3908.070713] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3908.070717] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3908.070721] do_raw_spin_lock+0x177/0x190 [ 3908.070725] _raw_spin_lock+0x21/0x30 [ 3908.070730] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3908.071303] obd_mod_free+0x46/0x140 [ptlrpc] [ 3908.071509] mdc_commit_open+0x91/0xb0 [mdc] [ 3908.071552] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3908.071746] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3908.071939] after_reply+0xe2c/0x1780 [ptlrpc] [ 3908.072432] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3908.072619] ? __next_timer_interrupt+0x160/0x160 [ 3908.072630] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3908.072830] ? do_wait_intr+0xf0/0xf0 [ 3908.072841] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3908.073142] kthread+0x1d1/0x200 [ 3908.073148] ? set_kthread_struct+0x70/0x70 [ 3908.073152] ret_from_fork+0x1f/0x30 [ 3908.210219] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3908.227908] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3908.232979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3908.237446] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3908.239976] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3908.250297] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3908.252941] RAX: 0000000000003b83 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3908.265988] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3908.272506] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3908.281631] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3908.288871] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3908.292463] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3908.297866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3908.299937] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3908.303456] Call Trace: [ 3908.304536] [ 3908.305561] ? show_regs.cold.9+0x22/0x2f [ 3908.307693] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3908.309861] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3908.311851] ? watchdog+0x40/0x40 [ 3908.313763] ? hrtimer_interrupt+0x128/0x2c0 [ 3908.315822] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3908.317889] ? apic_timer_interrupt+0xf/0x20 [ 3908.319944] [ 3908.320872] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3908.324979] do_raw_spin_lock+0x177/0x190 [ 3908.326872] _raw_spin_lock+0x21/0x30 [ 3908.334013] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3908.336106] ? do_raw_spin_unlock+0x75/0x190 [ 3908.338582] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3908.340820] ? do_wait_intr+0xf0/0xf0 [ 3908.342523] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3908.344947] mdc_close+0x53b/0xd70 [mdc] [ 3908.346696] lmv_close+0x18d/0x490 [lmv] [ 3908.351469] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3908.361234] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3908.368358] ll_md_close+0x274/0x800 [lustre] [ 3908.370266] ll_file_release+0x128/0xc50 [lustre] [ 3908.372736] __fput+0x10d/0x440 [ 3908.374238] ____fput+0x12/0x20 [ 3908.375632] task_work_run+0xc6/0x110 [ 3908.377107] exit_to_usermode_loop+0x1dd/0x1f0 [ 3908.379286] do_syscall_64+0x3d6/0x440 [ 3908.381205] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3908.383494] RIP: 0033:0x7f1a63983cd8 [ 3908.385167] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3908.395523] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3908.400458] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3908.406928] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3908.410392] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3908.413409] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3908.417661] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3912.053010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 3912.057464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3912.116606] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3912.129521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3912.139765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3912.146528] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3912.160930] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3912.166519] RAX: 00000000000010e6 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3912.178350] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3912.192628] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3912.201874] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3912.209420] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3912.216869] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3912.227644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3912.233822] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3912.240948] Call Trace: [ 3912.241898] [ 3912.242608] ? show_regs.cold.9+0x22/0x2f [ 3912.251596] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3912.256960] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3912.260922] ? watchdog+0x40/0x40 [ 3912.263866] ? hrtimer_interrupt+0x128/0x2c0 [ 3912.269506] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3912.274771] ? apic_timer_interrupt+0xf/0x20 [ 3912.276777] [ 3912.279479] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3912.283499] do_raw_spin_lock+0x177/0x190 [ 3912.289322] _raw_spin_lock+0x21/0x30 [ 3912.291071] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3912.299221] ? do_raw_spin_unlock+0x75/0x190 [ 3912.300532] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3912.305669] ? do_wait_intr+0xf0/0xf0 [ 3912.308220] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3912.311161] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3912.317965] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3912.321140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3912.323604] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3912.325669] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3912.335494] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3912.343535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3912.345587] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3912.353638] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3912.358562] lmv_intent_open+0x338/0x1150 [lmv] [ 3912.362800] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3912.369810] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3912.373952] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3912.376853] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3912.386376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3912.399419] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3912.408861] ? do_raw_spin_unlock+0x75/0x190 [ 3912.410365] ? _raw_spin_unlock+0x12/0x30 [ 3912.417380] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3912.423342] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3912.428469] ll_atomic_open+0x36a/0xd70 [lustre] [ 3912.434616] lookup_open+0x356/0xb40 [ 3912.436165] do_last+0x313/0xfc0 [ 3912.437395] ? nd_jump_root+0xe5/0x160 [ 3912.446620] ? path_init+0x437/0x520 [ 3912.454031] path_openat+0xf7/0x500 [ 3912.460134] ? do_raw_spin_unlock+0x75/0x190 [ 3912.462688] do_filp_open+0x99/0x140 [ 3912.476325] ? getname_flags+0x6e/0x330 [ 3912.477814] ? __check_object_size+0xff/0x256 [ 3912.487932] ? do_raw_spin_unlock+0x75/0x190 [ 3912.489467] ? _raw_spin_unlock+0x12/0x30 [ 3912.499248] do_sys_openat2+0x2b4/0x410 [ 3912.504634] do_sys_open+0x73/0xa0 [ 3912.506716] __x64_sys_openat+0x24/0x30 [ 3912.509726] do_syscall_64+0xc1/0x440 [ 3912.513437] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3912.515117] RIP: 0033:0x7f1a63983332 [ 3912.518451] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3912.540463] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3912.551726] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3912.559853] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3912.566696] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3912.576562] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3912.585669] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3916.772281] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1893s! [ 3916.779391] Showing busy workqueues and worker pools: [ 3916.790497] workqueue events: flags=0x0 [ 3916.797894] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3916.811716] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3916.816851] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3916.826633] pending: sec_gc_main [ptlrpc] [ 3916.833565] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3916.838008] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3916.857537] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3916.863387] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3916.875077] workqueue events_long: flags=0x0 [ 3916.884458] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3916.887847] pending: lu_objects_flush [obdclass] [ 3916.899281] workqueue events_unbound: flags=0x2 [ 3916.905712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3916.917628] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3916.937795] workqueue events_power_efficient: flags=0x80 [ 3916.942678] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3916.950112] pending: neigh_periodic_work, do_cache_clean [ 3916.955040] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3916.960518] pending: neigh_periodic_work, check_lifetime [ 3916.963182] workqueue mm_percpu_wq: flags=0x8 [ 3916.973515] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3916.976080] pending: vmstat_update [ 3916.987520] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3917.002679] pending: vmstat_update [ 3917.004121] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3917.015646] pending: vmstat_update [ 3917.021613] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3917.032343] pending: vmstat_update [ 3917.033839] workqueue writeback: flags=0x4a [ 3917.043476] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3917.054917] pending: wb_workfn, wb_workfn [ 3917.057504] workqueue xprtiod: flags=0xa [ 3917.061366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3917.068471] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3917.074550] workqueue ipv6_addrconf: flags=0x40008 [ 3917.081195] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3917.088589] pending: addrconf_verify_work [ 3917.091732] workqueue knbd0-recv: flags=0x1a [ 3917.103890] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3917.115019] in-flight: 647:recv_work [ 3917.117732] workqueue ptlrpc_pinger: flags=0x2 [ 3917.120518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3917.125206] pending: ptlrpc_pinger_main [ptlrpc] [ 3917.135473] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3907s workers=2 idle: 652 [ 3924.057037] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3924.060504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3924.079649] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3924.084359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3924.089541] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3924.092398] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3924.101219] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3924.105094] RAX: 000000000000687c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3924.108906] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3924.112540] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3924.115850] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3924.119358] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3924.122831] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3924.126813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3924.129606] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3924.132843] Call Trace: [ 3924.134099] [ 3924.134858] ? show_regs.cold.9+0x22/0x2f [ 3924.136531] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3924.139155] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3924.141248] ? watchdog+0x40/0x40 [ 3924.142622] ? hrtimer_interrupt+0x128/0x2c0 [ 3924.144843] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3924.151724] ? apic_timer_interrupt+0xf/0x20 [ 3924.153513] [ 3924.154378] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3924.156641] do_raw_spin_lock+0x177/0x190 [ 3924.157790] _raw_spin_lock+0x21/0x30 [ 3924.159135] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3924.160726] ? do_raw_spin_unlock+0x75/0x190 [ 3924.162052] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3924.163865] ? do_wait_intr+0xf0/0xf0 [ 3924.165481] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3924.167815] mdc_close+0x53b/0xd70 [mdc] [ 3924.169641] lmv_close+0x18d/0x490 [lmv] [ 3924.171262] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3924.173960] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3924.176000] ll_md_close+0x274/0x800 [lustre] [ 3924.177769] ll_file_release+0x128/0xc50 [lustre] [ 3924.180019] __fput+0x10d/0x440 [ 3924.181433] ____fput+0x12/0x20 [ 3924.183381] task_work_run+0xc6/0x110 [ 3924.185122] exit_to_usermode_loop+0x1dd/0x1f0 [ 3924.187191] do_syscall_64+0x3d6/0x440 [ 3924.188717] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3924.190404] RIP: 0033:0x7f1a63983cd8 [ 3924.191696] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3924.198686] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3924.201337] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3924.204099] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3924.206788] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3924.209299] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3924.211688] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3936.056161] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3936.058042] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3936.062342] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3936.062429] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3936.062436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3936.062439] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3936.062452] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3936.062456] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3936.062461] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3936.062464] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3936.062466] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3936.062468] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3936.062470] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3936.062473] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3936.062475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3936.062478] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3936.062487] Call Trace: [ 3936.062493] [ 3936.062499] ? show_regs.cold.9+0x22/0x2f [ 3936.062510] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3936.062517] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3936.062522] ? watchdog+0x40/0x40 [ 3936.062526] ? hrtimer_interrupt+0x128/0x2c0 [ 3936.062530] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3936.062537] ? apic_timer_interrupt+0xf/0x20 [ 3936.062542] [ 3936.062545] ? native_safe_halt+0x2e/0x50 [ 3936.062549] kvm_wait+0xcd/0xe0 [ 3936.062733] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3936.062741] do_raw_spin_lock+0x177/0x190 [ 3936.062814] _raw_spin_lock+0x21/0x30 [ 3936.062821] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3936.063316] ? do_raw_spin_unlock+0x75/0x190 [ 3936.063322] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3936.071484] Modules linked in: lustre(O) [ 3936.109862] ? do_wait_intr+0xf0/0xf0 [ 3936.120625] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw [ 3936.127531] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3936.130852] libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3936.130881] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3936.130889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3936.130892] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3936.130906] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3936.130911] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3936.130915] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3936.130917] RDX: 0000000000000ba4 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3936.130920] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3936.130922] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3936.130924] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3936.130927] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3936.130930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3936.130933] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3936.130942] Call Trace: [ 3936.130948] [ 3936.130952] ? show_regs.cold.9+0x22/0x2f [ 3936.130962] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3936.130968] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3936.131150] ? watchdog+0x40/0x40 [ 3936.131156] ? hrtimer_interrupt+0x128/0x2c0 [ 3936.131161] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3936.131171] ? apic_timer_interrupt+0xf/0x20 [ 3936.131176] [ 3936.131179] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3936.131184] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3936.131189] do_raw_spin_lock+0x177/0x190 [ 3936.131194] _raw_spin_lock+0x21/0x30 [ 3936.131200] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3936.131586] obd_mod_free+0x46/0x140 [ptlrpc] [ 3936.132167] mdc_commit_open+0x91/0xb0 [mdc] [ 3936.132204] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3936.152675] mdc_close+0x53b/0xd70 [mdc] [ 3936.152723] lmv_close+0x18d/0x490 [lmv] [ 3936.153109] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3936.153219] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3936.153301] ll_md_close+0x274/0x800 [lustre] [ 3936.153461] ll_file_release+0x128/0xc50 [lustre] [ 3936.153541] __fput+0x10d/0x440 [ 3936.160690] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3936.162152] after_reply+0xe2c/0x1780 [ptlrpc] [ 3936.162360] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3936.171346] ____fput+0x12/0x20 [ 3936.178941] ? __next_timer_interrupt+0x160/0x160 [ 3936.185195] task_work_run+0xc6/0x110 [ 3936.185216] exit_to_usermode_loop+0x1dd/0x1f0 [ 3936.185222] do_syscall_64+0x3d6/0x440 [ 3936.185227] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3936.185236] RIP: 0033:0x7f1a63983cd8 [ 3936.185244] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3936.185248] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3936.185253] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3936.185256] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3936.185258] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3936.185260] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3936.185263] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3936.504116] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3936.506564] ? do_wait_intr+0xf0/0xf0 [ 3936.508148] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3936.511163] kthread+0x1d1/0x200 [ 3936.512421] ? set_kthread_struct+0x70/0x70 [ 3936.514318] ret_from_fork+0x1f/0x30 [ 3944.055564] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3944.067290] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3944.174359] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3944.200120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3944.213064] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3944.215436] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3944.246991] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3944.251839] RAX: 0000000000004a19 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3944.255288] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3944.261732] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3944.274660] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3944.284310] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3944.302939] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3944.315717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3944.320322] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3944.331923] Call Trace: [ 3944.336880] [ 3944.338017] ? show_regs.cold.9+0x22/0x2f [ 3944.345749] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3944.353324] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3944.358229] ? watchdog+0x40/0x40 [ 3944.362231] ? hrtimer_interrupt+0x128/0x2c0 [ 3944.369067] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3944.374248] ? apic_timer_interrupt+0xf/0x20 [ 3944.383094] [ 3944.384929] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3944.396117] do_raw_spin_lock+0x177/0x190 [ 3944.404596] _raw_spin_lock+0x21/0x30 [ 3944.409912] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3944.418022] ? do_raw_spin_unlock+0x75/0x190 [ 3944.427784] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3944.436182] ? do_wait_intr+0xf0/0xf0 [ 3944.441837] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3944.449776] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3944.461137] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3944.470145] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3944.472931] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3944.475903] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3944.478254] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3944.483135] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3944.488193] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3944.495626] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3944.499006] lmv_intent_open+0x338/0x1150 [lmv] [ 3944.503350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3944.515741] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3944.521996] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3944.530722] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3944.542957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3944.551175] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3944.563089] ? do_raw_spin_unlock+0x75/0x190 [ 3944.575652] ? _raw_spin_unlock+0x12/0x30 [ 3944.585517] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3944.592863] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3944.596862] ll_atomic_open+0x36a/0xd70 [lustre] [ 3944.599806] lookup_open+0x356/0xb40 [ 3944.606631] do_last+0x313/0xfc0 [ 3944.611138] ? nd_jump_root+0xe5/0x160 [ 3944.613892] ? path_init+0x437/0x520 [ 3944.615555] path_openat+0xf7/0x500 [ 3944.623047] ? do_raw_spin_unlock+0x75/0x190 [ 3944.629544] do_filp_open+0x99/0x140 [ 3944.634470] ? getname_flags+0x6e/0x330 [ 3944.641401] ? __check_object_size+0xff/0x256 [ 3944.647811] ? do_raw_spin_unlock+0x75/0x190 [ 3944.662141] ? _raw_spin_unlock+0x12/0x30 [ 3944.666390] do_sys_openat2+0x2b4/0x410 [ 3944.674906] do_sys_open+0x73/0xa0 [ 3944.680853] __x64_sys_openat+0x24/0x30 [ 3944.689055] do_syscall_64+0xc1/0x440 [ 3944.690486] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3944.699240] RIP: 0033:0x7f1a63983332 [ 3944.705469] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3944.741174] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3944.743653] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3944.752375] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3944.765411] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3944.773139] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3944.785258] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3947.492481] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1924s! [ 3947.513382] Showing busy workqueues and worker pools: [ 3947.531016] workqueue events: flags=0x0 [ 3947.536950] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3947.557765] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3947.573061] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.586419] pending: sec_gc_main [ptlrpc] [ 3947.599757] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3947.615151] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3947.635980] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3947.645608] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3947.666891] workqueue events_long: flags=0x0 [ 3947.685699] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.693771] pending: lu_objects_flush [obdclass] [ 3947.703906] workqueue events_unbound: flags=0x2 [ 3947.716546] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3947.735995] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3947.774654] workqueue events_power_efficient: flags=0x80 [ 3947.777085] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3947.786616] pending: neigh_periodic_work, do_cache_clean [ 3947.790225] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3947.794478] pending: neigh_periodic_work, check_lifetime [ 3947.801416] workqueue mm_percpu_wq: flags=0x8 [ 3947.811894] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.814437] pending: vmstat_update [ 3947.818874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.830636] pending: vmstat_update [ 3947.834774] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.838497] pending: vmstat_update [ 3947.842100] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.850081] pending: vmstat_update [ 3947.855895] workqueue writeback: flags=0x4a [ 3947.861893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3947.870515] pending: wb_workfn, wb_workfn [ 3947.879875] workqueue xprtiod: flags=0xa [ 3947.887738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3947.892486] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3947.898092] workqueue ipv6_addrconf: flags=0x40008 [ 3947.902959] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3947.908570] pending: addrconf_verify_work [ 3947.912795] workqueue knbd0-recv: flags=0x1a [ 3947.914014] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3947.919107] in-flight: 647:recv_work [ 3947.921710] workqueue ptlrpc_pinger: flags=0x2 [ 3947.924849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3947.928909] pending: ptlrpc_pinger_main [ptlrpc] [ 3947.931943] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3937s workers=2 idle: 652 [ 3952.062661] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3952.082373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3952.143467] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3952.154966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3952.163804] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3952.179739] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3952.213842] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3952.226878] RAX: 0000000000002513 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3952.254948] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3952.267405] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3952.282537] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3952.293776] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3952.307706] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3952.323310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3952.338056] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3952.350846] Call Trace: [ 3952.353685] [ 3952.358821] ? show_regs.cold.9+0x22/0x2f [ 3952.366411] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3952.371036] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3952.379363] ? watchdog+0x40/0x40 [ 3952.380460] ? hrtimer_interrupt+0x128/0x2c0 [ 3952.384789] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3952.389763] ? apic_timer_interrupt+0xf/0x20 [ 3952.391650] [ 3952.392474] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3952.399387] do_raw_spin_lock+0x177/0x190 [ 3952.403941] _raw_spin_lock+0x21/0x30 [ 3952.405776] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3952.408297] ? do_raw_spin_unlock+0x75/0x190 [ 3952.409782] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3952.422497] ? do_wait_intr+0xf0/0xf0 [ 3952.426746] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3952.430563] mdc_close+0x53b/0xd70 [mdc] [ 3952.443125] lmv_close+0x18d/0x490 [lmv] [ 3952.447304] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3952.457747] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3952.463106] ll_md_close+0x274/0x800 [lustre] [ 3952.467186] ll_file_release+0x128/0xc50 [lustre] [ 3952.474618] __fput+0x10d/0x440 [ 3952.475917] ____fput+0x12/0x20 [ 3952.481313] task_work_run+0xc6/0x110 [ 3952.484913] exit_to_usermode_loop+0x1dd/0x1f0 [ 3952.490526] do_syscall_64+0x3d6/0x440 [ 3952.491899] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3952.496686] RIP: 0033:0x7f1a63983cd8 [ 3952.500194] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3952.534167] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3952.541060] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3952.546384] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3952.556322] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3952.563533] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3952.577186] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3964.056068] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3964.058019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3964.059890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 [ 3964.066396] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3964.066474] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3964.066491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3964.066494] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3964.066509] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3964.066513] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3964.066518] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3964.066520] RDX: 00000000000010ae RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3964.066523] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3964.066525] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3964.066527] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3964.066530] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3964.066533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3964.066535] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3964.066544] Call Trace: [ 3964.066550] [ 3964.066556] ? show_regs.cold.9+0x22/0x2f [ 3964.066566] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3964.066572] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3964.066576] ? watchdog+0x40/0x40 [ 3964.066581] ? hrtimer_interrupt+0x128/0x2c0 [ 3964.066585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3964.066593] ? apic_timer_interrupt+0xf/0x20 [ 3964.115982] squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3964.116016] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3964.116024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3964.116027] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3964.116043] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3964.116048] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3964.116052] RAX: 0000000000001387 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3964.116055] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3964.116057] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3964.116059] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3964.116062] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3964.116065] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3964.116069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3964.116071] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3964.116079] Call Trace: [ 3964.116085] [ 3964.116091] ? show_regs.cold.9+0x22/0x2f [ 3964.116100] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3964.116106] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3964.116111] ? watchdog+0x40/0x40 [ 3964.116115] ? hrtimer_interrupt+0x128/0x2c0 [ 3964.116120] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3964.116128] ? apic_timer_interrupt+0xf/0x20 [ 3964.116133] [ 3964.116136] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3964.116143] do_raw_spin_lock+0x177/0x190 [ 3964.116147] _raw_spin_lock+0x21/0x30 [ 3964.116152] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3964.117183] ? do_raw_spin_unlock+0x75/0x190 [ 3964.117192] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3964.118947] ? do_wait_intr+0xf0/0xf0 [ 3964.118958] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3964.119181] mdc_close+0x53b/0xd70 [mdc] [ 3964.119219] lmv_close+0x18d/0x490 [lmv] [ 3964.119252] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3964.120098] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3964.200767] [ 3964.200780] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3964.200795] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3964.200800] do_raw_spin_lock+0x177/0x190 [ 3964.200806] _raw_spin_lock+0x21/0x30 [ 3964.200814] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3964.202771] obd_mod_free+0x46/0x140 [ptlrpc] [ 3964.209228] ll_md_close+0x274/0x800 [lustre] [ 3964.209646] ll_file_release+0x128/0xc50 [lustre] [ 3964.209731] __fput+0x10d/0x440 [ 3964.209741] ____fput+0x12/0x20 [ 3964.209745] task_work_run+0xc6/0x110 [ 3964.209753] exit_to_usermode_loop+0x1dd/0x1f0 [ 3964.209760] do_syscall_64+0x3d6/0x440 [ 3964.209764] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3964.209773] RIP: 0033:0x7f1a63983cd8 [ 3964.209780] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3964.209784] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3964.209790] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3964.209792] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3964.209794] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3964.209796] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3964.209798] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3965.223541] mdc_commit_open+0x91/0xb0 [mdc] [ 3965.234262] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3965.244314] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3965.259729] after_reply+0xe2c/0x1780 [ptlrpc] [ 3965.266553] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3965.278130] ? __next_timer_interrupt+0x160/0x160 [ 3965.285561] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3965.291761] ? do_wait_intr+0xf0/0xf0 [ 3965.297421] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3965.301904] kthread+0x1d1/0x200 [ 3965.308249] ? set_kthread_struct+0x70/0x70 [ 3965.313799] ret_from_fork+0x1f/0x30 [ 3972.056535] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 3972.079258] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3972.182559] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3972.215414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3972.235531] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3972.248269] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3972.291032] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3972.302905] RAX: 000000000000179a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3972.314652] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 3972.324959] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 3972.342551] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 3972.366225] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 3972.377387] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 3972.394193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3972.407121] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 3972.409780] Call Trace: [ 3972.429783] [ 3972.430960] ? show_regs.cold.9+0x22/0x2f [ 3972.443385] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3972.451755] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3972.462612] ? watchdog+0x40/0x40 [ 3972.471200] ? hrtimer_interrupt+0x128/0x2c0 [ 3972.478480] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3972.480107] ? apic_timer_interrupt+0xf/0x20 [ 3972.492540] [ 3972.497004] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3972.506862] do_raw_spin_lock+0x177/0x190 [ 3972.508239] _raw_spin_lock+0x21/0x30 [ 3972.522674] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3972.530294] ? do_raw_spin_unlock+0x75/0x190 [ 3972.531959] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3972.551934] ? do_wait_intr+0xf0/0xf0 [ 3972.569725] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3972.571778] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3972.584606] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3972.593705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3972.599634] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3972.603366] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3972.608768] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3972.618825] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3972.623450] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3972.636601] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3972.641499] lmv_intent_open+0x338/0x1150 [lmv] [ 3972.646308] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3972.659715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3972.670107] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3972.676803] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3972.682812] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3972.691755] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3972.693503] ? do_raw_spin_unlock+0x75/0x190 [ 3972.710937] ? _raw_spin_unlock+0x12/0x30 [ 3972.716386] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3972.723854] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3972.731637] ll_atomic_open+0x36a/0xd70 [lustre] [ 3972.739727] lookup_open+0x356/0xb40 [ 3972.742514] do_last+0x313/0xfc0 [ 3972.744882] ? nd_jump_root+0xe5/0x160 [ 3972.747311] ? path_init+0x437/0x520 [ 3972.749493] path_openat+0xf7/0x500 [ 3972.751680] ? do_raw_spin_unlock+0x75/0x190 [ 3972.758720] do_filp_open+0x99/0x140 [ 3972.766674] ? getname_flags+0x6e/0x330 [ 3972.770505] ? __check_object_size+0xff/0x256 [ 3972.773630] ? do_raw_spin_unlock+0x75/0x190 [ 3972.777647] ? _raw_spin_unlock+0x12/0x30 [ 3972.790090] do_sys_openat2+0x2b4/0x410 [ 3972.791533] do_sys_open+0x73/0xa0 [ 3972.792802] __x64_sys_openat+0x24/0x30 [ 3972.805878] do_syscall_64+0xc1/0x440 [ 3972.807233] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3972.837307] RIP: 0033:0x7f1a63983332 [ 3972.839358] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3972.860730] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3972.873819] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 3972.880181] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 3972.882578] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 3972.888634] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 3972.896236] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 3978.208273] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1955s! [ 3978.213523] Showing busy workqueues and worker pools: [ 3978.216245] workqueue events: flags=0x0 [ 3978.220913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3978.228060] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 3978.233557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3978.237472] pending: sec_gc_main [ptlrpc] [ 3978.240491] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3978.244488] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 3978.251786] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3978.254843] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 3978.259286] workqueue events_long: flags=0x0 [ 3978.261742] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3978.266225] pending: lu_objects_flush [obdclass] [ 3978.269834] workqueue events_unbound: flags=0x2 [ 3978.272880] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3978.276809] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3978.283631] workqueue events_power_efficient: flags=0x80 [ 3978.286768] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3978.291211] pending: neigh_periodic_work, do_cache_clean [ 3978.295300] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3978.299950] pending: neigh_periodic_work, check_lifetime [ 3978.304387] workqueue mm_percpu_wq: flags=0x8 [ 3978.307207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3978.311502] pending: vmstat_update [ 3978.313329] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3978.318487] pending: vmstat_update [ 3978.320583] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3978.324515] pending: vmstat_update [ 3978.326816] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3978.330697] pending: vmstat_update [ 3978.333135] workqueue writeback: flags=0x4a [ 3978.335493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3978.338790] pending: wb_workfn, wb_workfn [ 3978.341278] workqueue xprtiod: flags=0xa [ 3978.343452] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3978.346275] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 3978.350213] workqueue ipv6_addrconf: flags=0x40008 [ 3978.352818] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3978.356498] pending: addrconf_verify_work [ 3978.358946] workqueue knbd0-recv: flags=0x1a [ 3978.362517] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3978.366323] in-flight: 647:recv_work [ 3978.368969] workqueue ptlrpc_pinger: flags=0x2 [ 3978.372087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3978.375097] pending: ptlrpc_pinger_main [ptlrpc] [ 3978.379330] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3968s workers=2 idle: 652 [ 3980.057021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 3980.061662] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3980.088143] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3980.094136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3980.098607] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3980.101840] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3980.115181] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3980.119933] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3980.124384] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 3980.128891] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 3980.133908] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 3980.139367] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 3980.143207] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 3980.148521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3980.151932] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 3980.156189] Call Trace: [ 3980.157293] [ 3980.158394] ? show_regs.cold.9+0x22/0x2f [ 3980.160957] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3980.164493] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3980.167252] ? watchdog+0x40/0x40 [ 3980.168923] ? hrtimer_interrupt+0x128/0x2c0 [ 3980.171528] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3980.176969] ? apic_timer_interrupt+0xf/0x20 [ 3980.179425] [ 3980.180532] ? native_safe_halt+0x2e/0x50 [ 3980.182695] kvm_wait+0xcd/0xe0 [ 3980.184716] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3980.188039] do_raw_spin_lock+0x177/0x190 [ 3980.189848] _raw_spin_lock+0x21/0x30 [ 3980.191576] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3980.194483] ? do_raw_spin_unlock+0x75/0x190 [ 3980.196419] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3980.199801] ? do_wait_intr+0xf0/0xf0 [ 3980.202594] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3980.205679] mdc_close+0x53b/0xd70 [mdc] [ 3980.208739] lmv_close+0x18d/0x490 [lmv] [ 3980.211464] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3980.214908] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3980.217862] ll_md_close+0x274/0x800 [lustre] [ 3980.221017] ll_file_release+0x128/0xc50 [lustre] [ 3980.225877] __fput+0x10d/0x440 [ 3980.227713] ____fput+0x12/0x20 [ 3980.230498] task_work_run+0xc6/0x110 [ 3980.232532] exit_to_usermode_loop+0x1dd/0x1f0 [ 3980.235814] do_syscall_64+0x3d6/0x440 [ 3980.239742] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3980.243240] RIP: 0033:0x7f1a63983cd8 [ 3980.247794] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3980.259987] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3980.264921] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 3980.271673] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3980.276706] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 3980.281432] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 3980.288860] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3992.056057] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 3992.058177] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 3992.066778] Modules linked in: lustre(O) [ 3992.070194] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3992.070349] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3992.070357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3992.070360] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3992.070379] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3992.070388] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3992.070393] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3992.070396] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 3992.070398] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 3992.070400] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 3992.070403] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3992.070406] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 3992.070409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3992.070411] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 3992.070419] Call Trace: [ 3992.070428] [ 3992.070435] ? show_regs.cold.9+0x22/0x2f [ 3992.070446] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3992.070457] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3992.070462] ? watchdog+0x40/0x40 [ 3992.070466] ? hrtimer_interrupt+0x128/0x2c0 [ 3992.070470] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3992.070477] ? apic_timer_interrupt+0xf/0x20 [ 3992.070481] [ 3992.070483] ? native_safe_halt+0x2e/0x50 [ 3992.070487] kvm_wait+0xcd/0xe0 [ 3992.070494] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3992.070506] do_raw_spin_lock+0x177/0x190 [ 3992.070509] _raw_spin_lock+0x21/0x30 [ 3992.070514] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3992.071018] obd_mod_free+0x46/0x140 [ptlrpc] [ 3992.076070] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3992.108623] mdc_commit_open+0x91/0xb0 [mdc] [ 3992.119809] [ 3992.125277] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3992.130755] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3992.144422] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3992.150961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3992.150971] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3992.150986] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3992.150990] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3992.150996] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 3992.151054] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 3992.151056] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3992.151174] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 3992.151177] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 3992.151180] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 3992.151183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3992.151186] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 3992.151194] Call Trace: [ 3992.151199] [ 3992.151268] ? show_regs.cold.9+0x22/0x2f [ 3992.151279] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3992.151285] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3992.151288] ? watchdog+0x40/0x40 [ 3992.151292] ? hrtimer_interrupt+0x128/0x2c0 [ 3992.151295] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3992.151301] ? apic_timer_interrupt+0xf/0x20 [ 3992.151305] [ 3992.151307] ? native_safe_halt+0x2e/0x50 [ 3992.151311] kvm_wait+0xcd/0xe0 [ 3992.151318] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3992.151326] do_raw_spin_lock+0x177/0x190 [ 3992.151330] _raw_spin_lock+0x21/0x30 [ 3992.151334] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3992.155019] after_reply+0xe2c/0x1780 [ptlrpc] [ 3992.155379] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3992.156172] ? __next_timer_interrupt+0x160/0x160 [ 3992.156184] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3992.156453] ? do_wait_intr+0xf0/0xf0 [ 3992.156463] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3992.156659] kthread+0x1d1/0x200 [ 3992.156666] ? set_kthread_struct+0x70/0x70 [ 3992.156670] ret_from_fork+0x1f/0x30 [ 3992.420058] ? do_raw_spin_unlock+0x75/0x190 [ 3992.424759] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3992.428475] ? do_wait_intr+0xf0/0xf0 [ 3992.430000] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3992.431493] mdc_close+0x53b/0xd70 [mdc] [ 3992.434198] lmv_close+0x18d/0x490 [lmv] [ 3992.438354] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3992.442750] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3992.445765] ll_md_close+0x274/0x800 [lustre] [ 3992.449005] ll_file_release+0x128/0xc50 [lustre] [ 3992.452876] __fput+0x10d/0x440 [ 3992.454986] ____fput+0x12/0x20 [ 3992.457025] task_work_run+0xc6/0x110 [ 3992.460342] exit_to_usermode_loop+0x1dd/0x1f0 [ 3992.462866] do_syscall_64+0x3d6/0x440 [ 3992.464543] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3992.467490] RIP: 0033:0x7f1a63983cd8 [ 3992.469993] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3992.480893] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3992.484711] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 3992.490996] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 3992.494396] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 3992.496594] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 3992.501863] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4004.053026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 4004.060718] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4004.101407] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4004.116827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4004.123693] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4004.125627] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4004.133628] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4004.137400] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4004.141842] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4004.145047] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4004.151410] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4004.159068] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4004.163996] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4004.168817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4004.171640] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4004.174790] Call Trace: [ 4004.175984] [ 4004.176808] ? show_regs.cold.9+0x22/0x2f [ 4004.178605] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4004.182886] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4004.187076] ? watchdog+0x40/0x40 [ 4004.188682] ? hrtimer_interrupt+0x128/0x2c0 [ 4004.191342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4004.194312] ? apic_timer_interrupt+0xf/0x20 [ 4004.196520] [ 4004.197241] ? native_safe_halt+0x2e/0x50 [ 4004.198498] kvm_wait+0xcd/0xe0 [ 4004.199717] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4004.201593] do_raw_spin_lock+0x177/0x190 [ 4004.203362] _raw_spin_lock+0x21/0x30 [ 4004.207138] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4004.210604] ? do_raw_spin_unlock+0x75/0x190 [ 4004.212807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4004.216988] ? do_wait_intr+0xf0/0xf0 [ 4004.221538] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4004.227967] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4004.235726] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4004.239517] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4004.243738] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4004.247803] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4004.252208] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4004.255253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4004.258933] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4004.262327] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4004.266961] lmv_intent_open+0x338/0x1150 [lmv] [ 4004.269653] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4004.273506] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4004.277522] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4004.279595] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4004.282457] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4004.285864] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4004.289476] ? do_raw_spin_unlock+0x75/0x190 [ 4004.292758] ? _raw_spin_unlock+0x12/0x30 [ 4004.296264] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4004.301109] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4004.304419] ll_atomic_open+0x36a/0xd70 [lustre] [ 4004.307845] lookup_open+0x356/0xb40 [ 4004.309802] do_last+0x313/0xfc0 [ 4004.312168] ? nd_jump_root+0xe5/0x160 [ 4004.317596] ? path_init+0x437/0x520 [ 4004.321785] path_openat+0xf7/0x500 [ 4004.326317] ? do_raw_spin_unlock+0x75/0x190 [ 4004.333342] do_filp_open+0x99/0x140 [ 4004.338219] ? getname_flags+0x6e/0x330 [ 4004.341015] ? __check_object_size+0xff/0x256 [ 4004.346403] ? do_raw_spin_unlock+0x75/0x190 [ 4004.349786] ? _raw_spin_unlock+0x12/0x30 [ 4004.354755] do_sys_openat2+0x2b4/0x410 [ 4004.356087] do_sys_open+0x73/0xa0 [ 4004.357410] __x64_sys_openat+0x24/0x30 [ 4004.359904] do_syscall_64+0xc1/0x440 [ 4004.360922] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4004.364881] RIP: 0033:0x7f1a63983332 [ 4004.366580] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4004.375755] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4004.379031] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4004.382514] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4004.387638] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4004.391997] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4004.403745] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4008.057034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4008.059599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4008.083916] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4008.090858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4008.095682] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4008.098073] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4008.106680] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4008.110285] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4008.113694] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4008.116692] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4008.119885] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4008.123840] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4008.126959] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4008.131195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4008.133812] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4008.137555] Call Trace: [ 4008.138715] [ 4008.139669] ? show_regs.cold.9+0x22/0x2f [ 4008.141872] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4008.144556] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4008.146720] ? watchdog+0x40/0x40 [ 4008.148406] ? hrtimer_interrupt+0x128/0x2c0 [ 4008.150289] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4008.152637] ? apic_timer_interrupt+0xf/0x20 [ 4008.154757] [ 4008.155927] ? native_safe_halt+0x2e/0x50 [ 4008.157743] kvm_wait+0xcd/0xe0 [ 4008.159226] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4008.161876] do_raw_spin_lock+0x177/0x190 [ 4008.163665] _raw_spin_lock+0x21/0x30 [ 4008.165694] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4008.167919] ? do_raw_spin_unlock+0x75/0x190 [ 4008.170638] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4008.173694] ? do_wait_intr+0xf0/0xf0 [ 4008.175258] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4008.178740] mdc_close+0x53b/0xd70 [mdc] [ 4008.180202] lmv_close+0x18d/0x490 [lmv] [ 4008.182208] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4008.185011] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4008.187787] ll_md_close+0x274/0x800 [lustre] [ 4008.190229] ll_file_release+0x128/0xc50 [lustre] [ 4008.192846] __fput+0x10d/0x440 [ 4008.199037] ____fput+0x12/0x20 [ 4008.200633] task_work_run+0xc6/0x110 [ 4008.203317] exit_to_usermode_loop+0x1dd/0x1f0 [ 4008.205665] do_syscall_64+0x3d6/0x440 [ 4008.208217] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4008.211431] RIP: 0033:0x7f1a63983cd8 [ 4008.213646] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4008.222534] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4008.225826] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4008.228921] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4008.232277] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4008.234912] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4008.237976] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4008.417065] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1985s! [ 4008.421194] Showing busy workqueues and worker pools: [ 4008.423760] workqueue events: flags=0x0 [ 4008.425801] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4008.429439] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4008.433354] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4008.437399] pending: sec_gc_main [ptlrpc] [ 4008.440176] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4008.443776] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4008.449639] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4008.452931] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4008.459082] workqueue events_long: flags=0x0 [ 4008.460881] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4008.463826] pending: lu_objects_flush [obdclass] [ 4008.466625] workqueue events_unbound: flags=0x2 [ 4008.468777] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4008.472418] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4008.476421] workqueue events_power_efficient: flags=0x80 [ 4008.478924] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4008.482764] pending: neigh_periodic_work, do_cache_clean [ 4008.485514] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4008.490432] pending: neigh_periodic_work, check_lifetime [ 4008.496564] workqueue mm_percpu_wq: flags=0x8 [ 4008.499605] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4008.503820] pending: vmstat_update [ 4008.507380] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4008.517741] pending: vmstat_update [ 4008.527571] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4008.532614] pending: vmstat_update [ 4008.535179] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4008.538688] pending: vmstat_update [ 4008.541201] workqueue writeback: flags=0x4a [ 4008.544471] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4008.547453] pending: wb_workfn, wb_workfn [ 4008.549609] workqueue xprtiod: flags=0xa [ 4008.552671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4008.557824] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4008.563696] workqueue ipv6_addrconf: flags=0x40008 [ 4008.567376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4008.571702] pending: addrconf_verify_work [ 4008.576488] workqueue knbd0-recv: flags=0x1a [ 4008.579566] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4008.584447] in-flight: 647:recv_work [ 4008.587153] workqueue ptlrpc_pinger: flags=0x2 [ 4008.591343] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4008.596340] pending: ptlrpc_pinger_main [ptlrpc] [ 4008.602526] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3998s workers=2 idle: 652 [ 4020.056156] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4020.058118] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4020.059149] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4020.059230] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4020.059238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4020.059240] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4020.059254] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4020.059258] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4020.059262] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4020.059265] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4020.059268] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4020.059270] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4020.059272] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4020.059276] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4020.059279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4020.059281] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4020.059291] Call Trace: [ 4020.059297] [ 4020.059303] ? show_regs.cold.9+0x22/0x2f [ 4020.059314] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4020.059320] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4020.059325] ? watchdog+0x40/0x40 [ 4020.059329] ? hrtimer_interrupt+0x128/0x2c0 [ 4020.059333] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4020.059339] ? apic_timer_interrupt+0xf/0x20 [ 4020.059344] [ 4020.059347] ? native_safe_halt+0x2e/0x50 [ 4020.059351] kvm_wait+0xcd/0xe0 [ 4020.059359] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4020.059367] do_raw_spin_lock+0x177/0x190 [ 4020.059372] _raw_spin_lock+0x21/0x30 [ 4020.059376] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4020.059652] ? do_raw_spin_unlock+0x75/0x190 [ 4020.059658] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4020.059867] ? do_wait_intr+0xf0/0xf0 [ 4020.059874] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4020.064004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4020.064092] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4020.095733] mdc_close+0x53b/0xd70 [mdc] [ 4020.095785] lmv_close+0x18d/0x490 [lmv] [ 4020.095815] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4020.096218] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4020.103532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4020.108114] ll_md_close+0x274/0x800 [lustre] [ 4020.110315] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4020.120466] ll_file_release+0x128/0xc50 [lustre] [ 4020.122991] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4020.128596] __fput+0x10d/0x440 [ 4020.131458] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 [ 4020.133916] ____fput+0x12/0x20 [ 4020.136685] ORIG_RAX: ffffffffffffff13 [ 4020.139523] task_work_run+0xc6/0x110 [ 4020.142503] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4020.144412] exit_to_usermode_loop+0x1dd/0x1f0 [ 4020.146752] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4020.147867] do_syscall_64+0x3d6/0x440 [ 4020.148659] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4020.150021] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4020.150155] RIP: 0033:0x7f1a63983cd8 [ 4020.151669] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4020.153294] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4020.154481] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4020.154487] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4020.154491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4020.154493] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4020.154500] Call Trace: [ 4020.155837] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 [ 4020.158352] [ 4020.159560] ORIG_RAX: 0000000000000003 [ 4020.160425] ? show_regs.cold.9+0x22/0x2f [ 4020.161612] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4020.162708] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4020.164498] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4020.166091] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4020.166106] ? watchdog+0x40/0x40 [ 4020.166113] ? hrtimer_interrupt+0x128/0x2c0 [ 4020.166117] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4020.167436] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4020.169540] ? apic_timer_interrupt+0xf/0x20 [ 4020.170880] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4020.170887] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4020.430096] [ 4020.432407] ? native_safe_halt+0x2e/0x50 [ 4020.437646] kvm_wait+0xcd/0xe0 [ 4020.441640] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4020.447629] do_raw_spin_lock+0x177/0x190 [ 4020.450668] _raw_spin_lock+0x21/0x30 [ 4020.452759] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4020.460610] obd_mod_free+0x46/0x140 [ptlrpc] [ 4020.466803] mdc_commit_open+0x91/0xb0 [mdc] [ 4020.477989] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4020.480715] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4020.483619] after_reply+0xe2c/0x1780 [ptlrpc] [ 4020.486581] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4020.489473] ? __next_timer_interrupt+0x160/0x160 [ 4020.491893] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4020.494576] ? do_wait_intr+0xf0/0xf0 [ 4020.496368] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4020.502723] kthread+0x1d1/0x200 [ 4020.506380] ? set_kthread_struct+0x70/0x70 [ 4020.508704] ret_from_fork+0x1f/0x30 [ 4036.053026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4036.056457] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4036.060264] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4036.102873] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4036.110988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4036.111068] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4036.111075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4036.111078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4036.111093] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4036.111097] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4036.111102] RAX: 0000000000006211 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4036.111105] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4036.111107] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4036.111109] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4036.111111] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4036.111115] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4036.111118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4036.111120] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4036.111130] Call Trace: [ 4036.111136] [ 4036.111142] ? show_regs.cold.9+0x22/0x2f [ 4036.111153] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4036.111160] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4036.111165] ? watchdog+0x40/0x40 [ 4036.111169] ? hrtimer_interrupt+0x128/0x2c0 [ 4036.111173] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4036.111181] ? apic_timer_interrupt+0xf/0x20 [ 4036.111186] [ 4036.111188] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4036.111195] do_raw_spin_lock+0x177/0x190 [ 4036.111200] _raw_spin_lock+0x21/0x30 [ 4036.111205] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4036.111644] ? do_raw_spin_unlock+0x75/0x190 [ 4036.111650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4036.122383] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4036.122393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4036.122409] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4036.122413] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4036.122419] RAX: 0000000000000586 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4036.122421] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4036.122423] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4036.122425] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4036.122427] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4036.122430] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4036.122433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4036.122436] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4036.122445] Call Trace: [ 4036.123686] [ 4036.150974] ? do_wait_intr+0xf0/0xf0 [ 4036.150995] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4036.151253] mdc_close+0x53b/0xd70 [mdc] [ 4036.151289] lmv_close+0x18d/0x490 [lmv] [ 4036.151322] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4036.151428] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4036.151589] ll_md_close+0x274/0x800 [lustre] [ 4036.151758] ll_file_release+0x128/0xc50 [lustre] [ 4036.151947] __fput+0x10d/0x440 [ 4036.151956] ____fput+0x12/0x20 [ 4036.151961] task_work_run+0xc6/0x110 [ 4036.151968] exit_to_usermode_loop+0x1dd/0x1f0 [ 4036.151975] do_syscall_64+0x3d6/0x440 [ 4036.151979] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4036.151988] RIP: 0033:0x7f1a63983cd8 [ 4036.151995] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4036.151999] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4036.152004] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4036.152007] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4036.152009] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4036.152011] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4036.152012] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4036.620732] ? show_regs.cold.9+0x22/0x2f [ 4036.623157] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4036.624490] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4036.626964] ? watchdog+0x40/0x40 [ 4036.628422] ? hrtimer_interrupt+0x128/0x2c0 [ 4036.630756] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4036.632701] ? apic_timer_interrupt+0xf/0x20 [ 4036.635105] [ 4036.636049] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4036.640190] do_raw_spin_lock+0x177/0x190 [ 4036.643889] _raw_spin_lock+0x21/0x30 [ 4036.645325] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4036.651237] ? do_raw_spin_unlock+0x75/0x190 [ 4036.655530] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4036.658397] ? do_wait_intr+0xf0/0xf0 [ 4036.663011] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4036.665373] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4036.667608] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4036.671306] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4036.673651] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4036.676676] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4036.679731] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4036.682415] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4036.685406] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4036.690641] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4036.694408] lmv_intent_open+0x338/0x1150 [lmv] [ 4036.697222] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4036.701604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4036.704709] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4036.707357] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4036.709585] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4036.714744] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4036.717389] ? do_raw_spin_unlock+0x75/0x190 [ 4036.719673] ? _raw_spin_unlock+0x12/0x30 [ 4036.722431] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4036.725237] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4036.731128] ll_atomic_open+0x36a/0xd70 [lustre] [ 4036.735938] lookup_open+0x356/0xb40 [ 4036.738425] do_last+0x313/0xfc0 [ 4036.740145] ? nd_jump_root+0xe5/0x160 [ 4036.743744] ? path_init+0x437/0x520 [ 4036.745541] path_openat+0xf7/0x500 [ 4036.749558] ? do_raw_spin_unlock+0x75/0x190 [ 4036.751389] do_filp_open+0x99/0x140 [ 4036.757102] ? getname_flags+0x6e/0x330 [ 4036.759095] ? __check_object_size+0xff/0x256 [ 4036.761032] ? do_raw_spin_unlock+0x75/0x190 [ 4036.768237] ? _raw_spin_unlock+0x12/0x30 [ 4036.770448] do_sys_openat2+0x2b4/0x410 [ 4036.778232] do_sys_open+0x73/0xa0 [ 4036.781994] __x64_sys_openat+0x24/0x30 [ 4036.786149] do_syscall_64+0xc1/0x440 [ 4036.796258] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4036.803500] RIP: 0033:0x7f1a63983332 [ 4036.808201] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4036.829678] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4036.845088] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4036.851512] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4036.872295] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4036.879819] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4036.890104] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4038.627632] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2015s! [ 4038.637956] Showing busy workqueues and worker pools: [ 4038.645874] workqueue events: flags=0x0 [ 4038.652864] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4038.659545] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4038.674694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4038.684939] pending: sec_gc_main [ptlrpc] [ 4038.691148] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4038.696506] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4038.707452] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4038.717384] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4038.723556] workqueue events_long: flags=0x0 [ 4038.738094] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4038.743297] pending: lu_objects_flush [obdclass] [ 4038.749292] workqueue events_unbound: flags=0x2 [ 4038.751113] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4038.761156] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4038.769568] workqueue events_power_efficient: flags=0x80 [ 4038.772659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4038.775172] pending: neigh_periodic_work, do_cache_clean [ 4038.777293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4038.779070] pending: neigh_periodic_work, check_lifetime [ 4038.783101] workqueue mm_percpu_wq: flags=0x8 [ 4038.785640] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4038.789389] pending: vmstat_update [ 4038.791432] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4038.795116] pending: vmstat_update [ 4038.797052] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4038.800412] pending: vmstat_update [ 4038.802236] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4038.809840] pending: vmstat_update [ 4038.813595] workqueue writeback: flags=0x4a [ 4038.816225] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4038.820324] pending: wb_workfn, wb_workfn [ 4038.824231] workqueue xprtiod: flags=0xa [ 4038.825415] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4038.830804] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4038.836601] workqueue ipv6_addrconf: flags=0x40008 [ 4038.839855] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4038.845846] pending: addrconf_verify_work [ 4038.848378] workqueue knbd0-recv: flags=0x1a [ 4038.851914] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4038.856245] in-flight: 647:recv_work [ 4038.858276] workqueue ptlrpc_pinger: flags=0x2 [ 4038.860543] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4038.863835] pending: ptlrpc_pinger_main [ptlrpc] [ 4038.866930] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4028s workers=2 idle: 652 [ 4048.057909] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4048.058605] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4048.071494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4048.071572] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4048.071580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4048.071582] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4048.071597] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4048.071601] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4048.071606] RAX: 0000000000000090 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4048.071608] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4048.071610] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4048.071613] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4048.071615] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4048.071618] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4048.071621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4048.071624] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4048.071633] Call Trace: [ 4048.071637] [ 4048.071643] ? show_regs.cold.9+0x22/0x2f [ 4048.071652] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4048.079527] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4048.079613] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4048.079621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4048.079624] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4048.079638] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4048.079642] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4048.079647] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4048.079649] RDX: 000000000000544f RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4048.079651] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4048.079652] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4048.079654] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4048.079657] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4048.079660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4048.079662] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4048.079679] Call Trace: [ 4048.079685] [ 4048.079691] ? show_regs.cold.9+0x22/0x2f [ 4048.079701] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4048.079708] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4048.079712] ? watchdog+0x40/0x40 [ 4048.079716] ? hrtimer_interrupt+0x128/0x2c0 [ 4048.079720] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4048.079730] ? apic_timer_interrupt+0xf/0x20 [ 4048.079734] [ 4048.079737] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4048.079742] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4048.079746] do_raw_spin_lock+0x177/0x190 [ 4048.079751] _raw_spin_lock+0x21/0x30 [ 4048.079756] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4048.080045] obd_mod_free+0x46/0x140 [ptlrpc] [ 4048.080458] mdc_commit_open+0x91/0xb0 [mdc] [ 4048.080494] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4048.081225] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4048.081532] after_reply+0xe2c/0x1780 [ptlrpc] [ 4048.133770] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4048.133788] ? watchdog+0x40/0x40 [ 4048.133796] ? hrtimer_interrupt+0x128/0x2c0 [ 4048.133800] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4048.133808] ? apic_timer_interrupt+0xf/0x20 [ 4048.133813] [ 4048.133818] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4048.133827] do_raw_spin_lock+0x177/0x190 [ 4048.133834] _raw_spin_lock+0x21/0x30 [ 4048.133839] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4048.134368] ? do_raw_spin_unlock+0x75/0x190 [ 4048.134374] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4048.134835] ? do_wait_intr+0xf0/0xf0 [ 4048.134843] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4048.155584] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4048.161260] mdc_close+0x53b/0xd70 [mdc] [ 4048.161310] lmv_close+0x18d/0x490 [lmv] [ 4048.161343] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4048.161460] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4048.161544] ll_md_close+0x274/0x800 [lustre] [ 4048.161632] ll_file_release+0x128/0xc50 [lustre] [ 4048.161906] __fput+0x10d/0x440 [ 4048.161916] ____fput+0x12/0x20 [ 4048.161921] task_work_run+0xc6/0x110 [ 4048.161929] exit_to_usermode_loop+0x1dd/0x1f0 [ 4048.161936] do_syscall_64+0x3d6/0x440 [ 4048.161941] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4048.161949] RIP: 0033:0x7f1a63983cd8 [ 4048.161958] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4048.161963] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4048.161969] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4048.161971] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4048.161973] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4048.161976] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4048.161978] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4048.522922] ? __next_timer_interrupt+0x160/0x160 [ 4048.524909] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4048.527425] ? do_wait_intr+0xf0/0xf0 [ 4048.528425] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4048.537286] kthread+0x1d1/0x200 [ 4048.541062] ? set_kthread_struct+0x70/0x70 [ 4048.543069] ret_from_fork+0x1f/0x30 [ 4062.458596] rcu: INFO: rcu_sched self-detected stall on CPU [ 4062.462104] rcu: 3-...!: (1215058 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 4062.471157] (t=2040058 jiffies g=269633 q=876) [ 4062.476323] rcu: rcu_sched kthread starved for 2039716 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 4062.490310] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4062.499250] rcu: RCU grace-period kthread stack dump: [ 4062.507934] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4062.521463] Call Trace: [ 4062.522402] __schedule+0x351/0xcb0 [ 4062.537171] schedule+0xc0/0x180 [ 4062.538519] schedule_timeout+0xb4/0x190 [ 4062.539976] ? do_raw_spin_unlock+0x75/0x190 [ 4062.544641] ? __next_timer_interrupt+0x160/0x160 [ 4062.548383] ? prepare_to_swait_event+0xf3/0x140 [ 4062.551794] rcu_gp_kthread+0x8dd/0x15b0 [ 4062.554995] ? force_qs_rnp+0x2c0/0x2c0 [ 4062.556902] kthread+0x1d1/0x200 [ 4062.558216] ? set_kthread_struct+0x70/0x70 [ 4062.562881] ret_from_fork+0x1f/0x30 [ 4062.565229] rcu: Stack dump where RCU GP kthread last ran: [ 4062.572625] Sending NMI from CPU 3 to CPUs 2: [ 4062.580686] NMI backtrace for cpu 2 [ 4062.580693] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4062.580696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4062.580698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4062.580700] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4062.580702] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 [ 4062.580705] RAX: 0000000000002a8b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4062.580706] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4062.580707] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4062.580709] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4062.580710] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4062.580712] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4062.580713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4062.580715] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4062.580720] Call Trace: [ 4062.580721] [ 4062.580722] ? show_regs.cold.9+0x22/0x2f [ 4062.580723] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4062.580724] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4062.580725] ? nmi_handle+0xbd/0x210 [ 4062.580726] ? default_do_nmi+0x4b/0x1b0 [ 4062.580727] ? do_nmi+0x280/0x380 [ 4062.580728] ? end_repeat_nmi+0x16/0x1f [ 4062.580729] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4062.580731] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4062.580732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4062.580733] [ 4062.580734] do_raw_spin_lock+0x177/0x190 [ 4062.580735] _raw_spin_lock+0x21/0x30 [ 4062.580736] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4062.580738] ? do_raw_spin_unlock+0x75/0x190 [ 4062.580739] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4062.580740] ? do_wait_intr+0xf0/0xf0 [ 4062.580741] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4062.580742] mdc_close+0x53b/0xd70 [mdc] [ 4062.580742] lmv_close+0x18d/0x490 [lmv] [ 4062.580743] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4062.580744] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4062.580745] ll_md_close+0x274/0x800 [lustre] [ 4062.580747] ll_file_release+0x128/0xc50 [lustre] [ 4062.580748] __fput+0x10d/0x440 [ 4062.580748] ____fput+0x12/0x20 [ 4062.580749] task_work_run+0xc6/0x110 [ 4062.580750] exit_to_usermode_loop+0x1dd/0x1f0 [ 4062.580751] do_syscall_64+0x3d6/0x440 [ 4062.580753] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4062.580754] RIP: 0033:0x7f1a63983cd8 [ 4062.580755] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4062.580757] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4062.580759] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4062.580761] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4062.580762] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4062.580763] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4062.580765] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4062.585135] NMI backtrace for cpu 3 [ 4062.908971] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4062.917345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4062.926112] Call Trace: [ 4062.927229] [ 4062.929459] ? dump_stack+0xbb/0x10e [ 4062.936728] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4062.944077] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4062.947118] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4062.953100] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4062.958910] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4062.962108] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4062.966957] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4062.971061] ? update_process_times+0x6d/0xc0 [ 4062.978506] ? tick_sched_handle.isra.18+0x27/0x90 [ 4062.984633] ? tick_sched_timer+0x53/0xb0 [ 4062.986085] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4062.990821] ? tick_sched_handle.isra.18+0x90/0x90 [ 4062.996923] ? hrtimer_interrupt+0x128/0x2c0 [ 4062.998996] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4063.002315] ? apic_timer_interrupt+0xf/0x20 [ 4063.005212] [ 4063.009102] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4063.012380] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4063.017294] ? do_raw_spin_lock+0x177/0x190 [ 4063.019643] ? _raw_spin_lock+0x21/0x30 [ 4063.023331] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4063.032391] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4063.039970] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4063.045241] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4063.051745] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4063.063844] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4063.068870] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4063.074899] ? __next_timer_interrupt+0x160/0x160 [ 4063.077957] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4063.083195] ? do_wait_intr+0xf0/0xf0 [ 4063.090300] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4063.094767] ? kthread+0x1d1/0x200 [ 4063.097369] ? set_kthread_struct+0x70/0x70 [ 4063.102887] ? ret_from_fork+0x1f/0x30 [ 4064.054447] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4064.064614] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4064.106940] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4064.115371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4064.123327] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4064.127675] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4064.139423] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4064.143796] RAX: 00000000000049b8 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4064.150157] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4064.153651] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4064.159421] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4064.161856] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4064.167769] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4064.180990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4064.188804] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4064.196662] Call Trace: [ 4064.198163] [ 4064.199232] ? show_regs.cold.9+0x22/0x2f [ 4064.204308] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4064.206273] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4064.217009] ? watchdog+0x40/0x40 [ 4064.225414] ? hrtimer_interrupt+0x128/0x2c0 [ 4064.227010] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4064.234395] ? apic_timer_interrupt+0xf/0x20 [ 4064.237266] [ 4064.241707] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4064.244196] do_raw_spin_lock+0x177/0x190 [ 4064.248248] _raw_spin_lock+0x21/0x30 [ 4064.255859] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4064.263634] ? do_raw_spin_unlock+0x75/0x190 [ 4064.269480] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4064.279338] ? do_wait_intr+0xf0/0xf0 [ 4064.281091] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4064.289905] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4064.299066] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4064.310266] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4064.316433] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4064.322825] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4064.327694] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4064.340757] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4064.343964] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4064.349057] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4064.356216] lmv_intent_open+0x338/0x1150 [lmv] [ 4064.361069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4064.367161] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4064.374008] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4064.379943] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4064.381716] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4064.389368] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4064.394055] ? do_raw_spin_unlock+0x75/0x190 [ 4064.398471] ? _raw_spin_unlock+0x12/0x30 [ 4064.402280] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4064.406069] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4064.414857] ll_atomic_open+0x36a/0xd70 [lustre] [ 4064.422351] lookup_open+0x356/0xb40 [ 4064.423629] do_last+0x313/0xfc0 [ 4064.428764] ? nd_jump_root+0xe5/0x160 [ 4064.435298] ? path_init+0x437/0x520 [ 4064.437851] path_openat+0xf7/0x500 [ 4064.439215] ? do_raw_spin_unlock+0x75/0x190 [ 4064.446987] do_filp_open+0x99/0x140 [ 4064.450041] ? getname_flags+0x6e/0x330 [ 4064.451107] ? __check_object_size+0xff/0x256 [ 4064.452450] ? do_raw_spin_unlock+0x75/0x190 [ 4064.460245] ? _raw_spin_unlock+0x12/0x30 [ 4064.463207] do_sys_openat2+0x2b4/0x410 [ 4064.468296] do_sys_open+0x73/0xa0 [ 4064.470239] __x64_sys_openat+0x24/0x30 [ 4064.474079] do_syscall_64+0xc1/0x440 [ 4064.481942] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4064.484902] RIP: 0033:0x7f1a63983332 [ 4064.486248] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4064.515180] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4064.527666] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4064.533122] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4064.537040] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4064.541197] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4064.552858] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4069.352367] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2046s! [ 4069.364534] Showing busy workqueues and worker pools: [ 4069.368337] workqueue events: flags=0x0 [ 4069.370389] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4069.375042] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4069.381017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4069.384156] pending: sec_gc_main [ptlrpc] [ 4069.386626] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4069.390946] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4069.396665] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4069.399651] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4069.402433] workqueue events_long: flags=0x0 [ 4069.410352] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4069.414603] pending: lu_objects_flush [obdclass] [ 4069.420310] workqueue events_unbound: flags=0x2 [ 4069.425835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4069.430211] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4069.437985] workqueue events_power_efficient: flags=0x80 [ 4069.440762] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4069.446034] pending: neigh_periodic_work, do_cache_clean [ 4069.453443] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4069.463094] pending: neigh_periodic_work, check_lifetime [ 4069.475512] workqueue mm_percpu_wq: flags=0x8 [ 4069.477138] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4069.486330] pending: vmstat_update [ 4069.489827] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4069.498593] pending: vmstat_update [ 4069.503074] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4069.512160] pending: vmstat_update [ 4069.517134] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4069.534630] pending: vmstat_update [ 4069.539919] workqueue writeback: flags=0x4a [ 4069.546279] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4069.554235] pending: wb_workfn, wb_workfn [ 4069.555881] workqueue xprtiod: flags=0xa [ 4069.568612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4069.582176] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4069.591149] workqueue ipv6_addrconf: flags=0x40008 [ 4069.597200] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4069.606896] pending: addrconf_verify_work [ 4069.608657] workqueue knbd0-recv: flags=0x1a [ 4069.612712] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4069.623116] in-flight: 647:recv_work [ 4069.626233] workqueue ptlrpc_pinger: flags=0x2 [ 4069.630167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4069.640060] pending: ptlrpc_pinger_main [ptlrpc] [ 4069.646187] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4059s workers=2 idle: 652 [ 4076.057881] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4076.063054] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4076.108024] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4076.119246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4076.124227] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4076.131853] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4076.158313] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4076.165548] RAX: 0000000000007e9b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4076.172158] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4076.180199] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4076.182834] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4076.193266] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4076.198062] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4076.204832] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4076.208986] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4076.217070] Call Trace: [ 4076.218686] [ 4076.220778] ? show_regs.cold.9+0x22/0x2f [ 4076.226810] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4076.232735] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4076.236259] ? watchdog+0x40/0x40 [ 4076.242978] ? hrtimer_interrupt+0x128/0x2c0 [ 4076.246613] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4076.259204] ? apic_timer_interrupt+0xf/0x20 [ 4076.261922] [ 4076.262817] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4076.267761] do_raw_spin_lock+0x177/0x190 [ 4076.269801] _raw_spin_lock+0x21/0x30 [ 4076.274083] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4076.277860] ? do_raw_spin_unlock+0x75/0x190 [ 4076.280521] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4076.284722] ? do_wait_intr+0xf0/0xf0 [ 4076.286076] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4076.293180] mdc_close+0x53b/0xd70 [mdc] [ 4076.300283] lmv_close+0x18d/0x490 [lmv] [ 4076.301734] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4076.312699] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4076.318922] ll_md_close+0x274/0x800 [lustre] [ 4076.323110] ll_file_release+0x128/0xc50 [lustre] [ 4076.335574] __fput+0x10d/0x440 [ 4076.336778] ____fput+0x12/0x20 [ 4076.337856] task_work_run+0xc6/0x110 [ 4076.357966] exit_to_usermode_loop+0x1dd/0x1f0 [ 4076.361801] do_syscall_64+0x3d6/0x440 [ 4076.369933] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4076.377190] RIP: 0033:0x7f1a63983cd8 [ 4076.380150] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4076.398721] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4076.404471] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4076.406968] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4076.417537] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4076.429793] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4076.444270] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4088.058172] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 4088.063701] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4088.071655] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4088.071731] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4088.071739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4088.071742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4088.071756] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4088.071761] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4088.071765] RAX: 0000000000007e85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4088.071767] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4088.071770] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4088.071772] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4088.071773] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4088.071776] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4088.071779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4088.071781] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4088.071791] Call Trace: [ 4088.071796] [ 4088.071801] ? show_regs.cold.9+0x22/0x2f [ 4088.071812] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4088.071818] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4088.071822] ? watchdog+0x40/0x40 [ 4088.071828] ? hrtimer_interrupt+0x128/0x2c0 [ 4088.071833] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4088.071841] ? apic_timer_interrupt+0xf/0x20 [ 4088.071846] [ 4088.071849] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4088.071856] do_raw_spin_lock+0x177/0x190 [ 4088.071860] _raw_spin_lock+0x21/0x30 [ 4088.071865] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4088.072140] ? do_raw_spin_unlock+0x75/0x190 [ 4088.072145] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4088.074269] ? do_wait_intr+0xf0/0xf0 [ 4088.078666] Modules linked in: lustre(O) [ 4088.104954] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4088.105735] mdc_close+0x53b/0xd70 [mdc] [ 4088.105788] lmv_close+0x18d/0x490 [lmv] [ 4088.105826] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4088.105960] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4088.106042] ll_md_close+0x274/0x800 [lustre] [ 4088.106124] ll_file_release+0x128/0xc50 [lustre] [ 4088.106351] __fput+0x10d/0x440 [ 4088.106361] ____fput+0x12/0x20 [ 4088.106366] task_work_run+0xc6/0x110 [ 4088.106380] exit_to_usermode_loop+0x1dd/0x1f0 [ 4088.106463] do_syscall_64+0x3d6/0x440 [ 4088.106468] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4088.106558] RIP: 0033:0x7f1a63983cd8 [ 4088.106567] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4088.106572] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4088.106577] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4088.106580] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4088.106582] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4088.106583] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4088.106586] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4088.391066] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4088.425485] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4088.437657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4088.444480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4088.450243] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4088.466154] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4088.477116] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4088.485721] RDX: 0000000000003f37 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4088.491708] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4088.499518] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4088.504200] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4088.513160] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4088.517431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4088.526707] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4088.532461] Call Trace: [ 4088.535022] [ 4088.536211] ? show_regs.cold.9+0x22/0x2f [ 4088.538475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4088.542119] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4088.545715] ? watchdog+0x40/0x40 [ 4088.547013] ? hrtimer_interrupt+0x128/0x2c0 [ 4088.548398] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4088.555836] ? apic_timer_interrupt+0xf/0x20 [ 4088.560080] [ 4088.561511] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4088.565445] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4088.573383] do_raw_spin_lock+0x177/0x190 [ 4088.574847] _raw_spin_lock+0x21/0x30 [ 4088.577801] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4088.582015] obd_mod_free+0x46/0x140 [ptlrpc] [ 4088.585990] mdc_commit_open+0x91/0xb0 [mdc] [ 4088.588261] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4088.596757] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4088.604634] after_reply+0xe2c/0x1780 [ptlrpc] [ 4088.607007] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4088.608949] ? __next_timer_interrupt+0x160/0x160 [ 4088.612419] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4088.620633] ? do_wait_intr+0xf0/0xf0 [ 4088.627725] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4088.636765] kthread+0x1d1/0x200 [ 4088.644140] ? set_kthread_struct+0x70/0x70 [ 4088.646709] ret_from_fork+0x1f/0x30 [ 4096.053363] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4096.058938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4096.106855] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4096.115876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4096.124592] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4096.128134] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4096.145011] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4096.151196] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4096.170133] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4096.182775] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4096.190077] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4096.205916] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4096.212029] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4096.217666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4096.223327] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4096.225664] Call Trace: [ 4096.237191] [ 4096.238040] ? show_regs.cold.9+0x22/0x2f [ 4096.240179] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4096.245881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4096.247793] ? watchdog+0x40/0x40 [ 4096.251422] ? hrtimer_interrupt+0x128/0x2c0 [ 4096.253423] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4096.256127] ? apic_timer_interrupt+0xf/0x20 [ 4096.258585] [ 4096.259776] ? native_safe_halt+0x2e/0x50 [ 4096.261062] kvm_wait+0xcd/0xe0 [ 4096.262056] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4096.264402] do_raw_spin_lock+0x177/0x190 [ 4096.266305] _raw_spin_lock+0x21/0x30 [ 4096.268117] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4096.270849] ? do_raw_spin_unlock+0x75/0x190 [ 4096.272557] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4096.275126] ? do_wait_intr+0xf0/0xf0 [ 4096.276742] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4096.279279] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4096.281756] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4096.284379] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4096.286594] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4096.289307] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4096.291421] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4096.293459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4096.295877] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4096.298891] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4096.301594] lmv_intent_open+0x338/0x1150 [lmv] [ 4096.303813] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4096.306606] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4096.309830] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4096.311868] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4096.314543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4096.317286] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4096.319510] ? do_raw_spin_unlock+0x75/0x190 [ 4096.321732] ? _raw_spin_unlock+0x12/0x30 [ 4096.323543] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4096.326137] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4096.328410] ll_atomic_open+0x36a/0xd70 [lustre] [ 4096.330890] lookup_open+0x356/0xb40 [ 4096.332709] do_last+0x313/0xfc0 [ 4096.334062] ? nd_jump_root+0xe5/0x160 [ 4096.335850] ? path_init+0x437/0x520 [ 4096.337855] path_openat+0xf7/0x500 [ 4096.339967] ? do_raw_spin_unlock+0x75/0x190 [ 4096.341786] do_filp_open+0x99/0x140 [ 4096.343349] ? getname_flags+0x6e/0x330 [ 4096.345183] ? __check_object_size+0xff/0x256 [ 4096.347458] ? do_raw_spin_unlock+0x75/0x190 [ 4096.349385] ? _raw_spin_unlock+0x12/0x30 [ 4096.351316] do_sys_openat2+0x2b4/0x410 [ 4096.352848] do_sys_open+0x73/0xa0 [ 4096.354445] __x64_sys_openat+0x24/0x30 [ 4096.356396] do_syscall_64+0xc1/0x440 [ 4096.357912] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4096.362949] RIP: 0033:0x7f1a63983332 [ 4096.366683] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4096.378961] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4096.382212] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4096.385040] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4096.390066] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4096.394624] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4096.398142] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4100.066212] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2077s! [ 4100.076978] Showing busy workqueues and worker pools: [ 4100.080792] workqueue events: flags=0x0 [ 4100.084070] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4100.087008] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4100.092099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4100.097231] pending: sec_gc_main [ptlrpc] [ 4100.099135] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4100.110276] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4100.116783] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4100.121535] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4100.129672] workqueue events_long: flags=0x0 [ 4100.133629] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4100.136136] pending: lu_objects_flush [obdclass] [ 4100.142703] workqueue events_unbound: flags=0x2 [ 4100.145182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4100.149615] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4100.155990] workqueue events_power_efficient: flags=0x80 [ 4100.157630] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4100.159743] pending: neigh_periodic_work, do_cache_clean [ 4100.161529] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4100.163711] pending: neigh_periodic_work, check_lifetime [ 4100.168091] workqueue mm_percpu_wq: flags=0x8 [ 4100.169878] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4100.174882] pending: vmstat_update [ 4100.176076] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4100.178357] pending: vmstat_update [ 4100.179485] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4100.181585] pending: vmstat_update [ 4100.182603] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4100.184827] pending: vmstat_update [ 4100.187984] workqueue writeback: flags=0x4a [ 4100.189645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4100.193064] pending: wb_workfn, wb_workfn [ 4100.197575] workqueue xprtiod: flags=0xa [ 4100.199929] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4100.203897] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4100.209316] workqueue ipv6_addrconf: flags=0x40008 [ 4100.211937] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4100.215416] pending: addrconf_verify_work [ 4100.217373] workqueue knbd0-recv: flags=0x1a [ 4100.218991] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4100.222381] in-flight: 647:recv_work [ 4100.224598] workqueue ptlrpc_pinger: flags=0x2 [ 4100.226603] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4100.228285] pending: ptlrpc_pinger_main [ptlrpc] [ 4100.231991] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4090s workers=2 idle: 652 [ 4104.058543] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4104.067183] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4104.092494] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4104.096740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4104.100041] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.102403] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4104.109657] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4104.112555] RAX: 00000000000046bc RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4104.114861] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4104.120585] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4104.124701] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4104.130751] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4104.133813] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4104.137707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4104.142531] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4104.147764] Call Trace: [ 4104.149014] [ 4104.150033] ? show_regs.cold.9+0x22/0x2f [ 4104.152075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4104.154678] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4104.157439] ? watchdog+0x40/0x40 [ 4104.159064] ? hrtimer_interrupt+0x128/0x2c0 [ 4104.161180] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4104.163794] ? apic_timer_interrupt+0xf/0x20 [ 4104.166245] [ 4104.168532] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.172684] do_raw_spin_lock+0x177/0x190 [ 4104.174807] _raw_spin_lock+0x21/0x30 [ 4104.176687] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4104.180534] ? do_raw_spin_unlock+0x75/0x190 [ 4104.183199] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4104.187512] ? do_wait_intr+0xf0/0xf0 [ 4104.189069] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4104.194498] mdc_close+0x53b/0xd70 [mdc] [ 4104.200229] lmv_close+0x18d/0x490 [lmv] [ 4104.202914] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4104.207472] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4104.210601] ll_md_close+0x274/0x800 [lustre] [ 4104.213512] ll_file_release+0x128/0xc50 [lustre] [ 4104.216883] __fput+0x10d/0x440 [ 4104.218134] ____fput+0x12/0x20 [ 4104.219967] task_work_run+0xc6/0x110 [ 4104.222025] exit_to_usermode_loop+0x1dd/0x1f0 [ 4104.226208] do_syscall_64+0x3d6/0x440 [ 4104.229040] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4104.231944] RIP: 0033:0x7f1a63983cd8 [ 4104.233833] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4104.241721] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4104.244871] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4104.247840] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4104.250038] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4104.253297] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4104.256798] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4116.057069] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4116.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4116.067574] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4116.067655] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4116.067663] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4116.067666] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4116.067681] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4116.067685] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4116.067690] RAX: 00000000000015e2 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4116.067692] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4116.067695] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4116.067698] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4116.067700] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4116.067703] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4116.067706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4116.067708] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4116.067717] Call Trace: [ 4116.067723] [ 4116.067730] ? show_regs.cold.9+0x22/0x2f [ 4116.067740] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4116.067746] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4116.067750] ? watchdog+0x40/0x40 [ 4116.067754] ? hrtimer_interrupt+0x128/0x2c0 [ 4116.067758] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4116.067766] ? apic_timer_interrupt+0xf/0x20 [ 4116.067771] [ 4116.067774] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4116.067779] do_raw_spin_lock+0x177/0x190 [ 4116.067785] _raw_spin_lock+0x21/0x30 [ 4116.067790] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4116.068589] ? do_raw_spin_unlock+0x75/0x190 [ 4116.068596] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4116.068797] ? do_wait_intr+0xf0/0xf0 [ 4116.068806] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4116.069240] mdc_close+0x53b/0xd70 [mdc] [ 4116.069297] lmv_close+0x18d/0x490 [lmv] [ 4116.069337] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4116.069464] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4116.069549] ll_md_close+0x274/0x800 [lustre] [ 4116.069669] ll_file_release+0x128/0xc50 [lustre] [ 4116.069754] __fput+0x10d/0x440 [ 4116.069763] ____fput+0x12/0x20 [ 4116.069767] task_work_run+0xc6/0x110 [ 4116.069774] exit_to_usermode_loop+0x1dd/0x1f0 [ 4116.069781] do_syscall_64+0x3d6/0x440 [ 4116.069867] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4116.069874] RIP: 0033:0x7f1a63983cd8 [ 4116.069881] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4116.069885] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4116.069891] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4116.069893] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4116.069895] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4116.069898] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4116.069900] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4116.297742] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4116.318538] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4116.322205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4116.325636] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4116.329872] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4116.337100] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4116.339500] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4116.341623] RDX: 00000000000057d8 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4116.345128] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4116.347337] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4116.349602] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4116.352042] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4116.361619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4116.368530] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4116.374357] Call Trace: [ 4116.374885] [ 4116.377910] ? show_regs.cold.9+0x22/0x2f [ 4116.379571] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4116.382320] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4116.383916] ? watchdog+0x40/0x40 [ 4116.385984] ? hrtimer_interrupt+0x128/0x2c0 [ 4116.388982] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4116.393843] ? apic_timer_interrupt+0xf/0x20 [ 4116.395576] [ 4116.396476] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4116.397990] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4116.402596] do_raw_spin_lock+0x177/0x190 [ 4116.403757] _raw_spin_lock+0x21/0x30 [ 4116.405344] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4116.408570] obd_mod_free+0x46/0x140 [ptlrpc] [ 4116.411301] mdc_commit_open+0x91/0xb0 [mdc] [ 4116.412453] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4116.417469] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4116.419861] after_reply+0xe2c/0x1780 [ptlrpc] [ 4116.421793] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4116.424100] ? __next_timer_interrupt+0x160/0x160 [ 4116.425657] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4116.427394] ? do_wait_intr+0xf0/0xf0 [ 4116.428504] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4116.430002] kthread+0x1d1/0x200 [ 4116.431365] ? set_kthread_struct+0x70/0x70 [ 4116.432548] ret_from_fork+0x1f/0x30 [ 4128.055243] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4128.060485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4128.096642] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4128.108607] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4128.115833] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4128.119071] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4128.132751] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4128.139174] RAX: 0000000000007c0c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4128.147364] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4128.157777] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4128.165167] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4128.169634] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4128.184646] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4128.199397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4128.208879] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4128.219492] Call Trace: [ 4128.222098] [ 4128.223467] ? show_regs.cold.9+0x22/0x2f [ 4128.227751] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4128.232837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4128.237290] ? watchdog+0x40/0x40 [ 4128.239676] ? hrtimer_interrupt+0x128/0x2c0 [ 4128.242245] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4128.245310] ? apic_timer_interrupt+0xf/0x20 [ 4128.248383] [ 4128.249224] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4128.250850] do_raw_spin_lock+0x177/0x190 [ 4128.257753] _raw_spin_lock+0x21/0x30 [ 4128.261287] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4128.269797] ? do_raw_spin_unlock+0x75/0x190 [ 4128.278451] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4128.285215] ? do_wait_intr+0xf0/0xf0 [ 4128.288468] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4128.292966] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4128.299981] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4128.308960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4128.316203] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4128.319143] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4128.323465] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4128.330162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4128.344411] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4128.353562] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4128.358555] lmv_intent_open+0x338/0x1150 [lmv] [ 4128.366637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4128.377238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4128.381832] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4128.387591] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4128.391759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4128.397653] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4128.403523] ? do_raw_spin_unlock+0x75/0x190 [ 4128.410775] ? _raw_spin_unlock+0x12/0x30 [ 4128.414547] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4128.426496] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4128.433282] ll_atomic_open+0x36a/0xd70 [lustre] [ 4128.445248] lookup_open+0x356/0xb40 [ 4128.449457] do_last+0x313/0xfc0 [ 4128.455687] ? nd_jump_root+0xe5/0x160 [ 4128.466529] ? path_init+0x437/0x520 [ 4128.469026] path_openat+0xf7/0x500 [ 4128.470298] ? do_raw_spin_unlock+0x75/0x190 [ 4128.486635] do_filp_open+0x99/0x140 [ 4128.487823] ? getname_flags+0x6e/0x330 [ 4128.489647] ? __check_object_size+0xff/0x256 [ 4128.491910] ? do_raw_spin_unlock+0x75/0x190 [ 4128.493629] ? _raw_spin_unlock+0x12/0x30 [ 4128.498382] do_sys_openat2+0x2b4/0x410 [ 4128.511287] do_sys_open+0x73/0xa0 [ 4128.512310] __x64_sys_openat+0x24/0x30 [ 4128.527542] do_syscall_64+0xc1/0x440 [ 4128.536270] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4128.540062] RIP: 0033:0x7f1a63983332 [ 4128.543105] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4128.569461] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4128.573549] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4128.580715] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4128.585457] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4128.591720] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4128.599770] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4130.272854] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2107s! [ 4130.275994] Showing busy workqueues and worker pools: [ 4130.282227] workqueue events: flags=0x0 [ 4130.287577] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4130.293365] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4130.296027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.299190] pending: sec_gc_main [ptlrpc] [ 4130.302838] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4130.306621] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4130.313086] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4130.316503] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4130.321814] workqueue events_long: flags=0x0 [ 4130.324037] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.329519] pending: lu_objects_flush [obdclass] [ 4130.342767] workqueue events_unbound: flags=0x2 [ 4130.344148] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4130.355606] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4130.373764] workqueue events_power_efficient: flags=0x80 [ 4130.380748] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4130.393827] pending: neigh_periodic_work, do_cache_clean [ 4130.396449] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4130.408702] pending: neigh_periodic_work, check_lifetime [ 4130.429344] workqueue mm_percpu_wq: flags=0x8 [ 4130.432819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.444771] pending: vmstat_update [ 4130.453113] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.460293] pending: vmstat_update [ 4130.462533] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.466476] pending: vmstat_update [ 4130.470916] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.486378] pending: vmstat_update [ 4130.487736] workqueue writeback: flags=0x4a [ 4130.489521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4130.498830] pending: wb_workfn, wb_workfn [ 4130.502323] workqueue xprtiod: flags=0xa [ 4130.508509] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4130.512296] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4130.516514] workqueue ipv6_addrconf: flags=0x40008 [ 4130.518956] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4130.522561] pending: addrconf_verify_work [ 4130.527124] workqueue knbd0-recv: flags=0x1a [ 4130.529821] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4130.534951] in-flight: 647:recv_work [ 4130.539021] workqueue ptlrpc_pinger: flags=0x2 [ 4130.544721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4130.551836] pending: ptlrpc_pinger_main [ptlrpc] [ 4130.556147] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4120s workers=2 idle: 652 [ 4132.056028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4132.059463] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4132.083465] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4132.089121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4132.094818] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4132.097570] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4132.107379] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4132.111638] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4132.115503] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4132.119382] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4132.123274] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4132.126986] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4132.130475] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4132.134996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4132.138037] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4132.141694] Call Trace: [ 4132.143277] [ 4132.144348] ? show_regs.cold.9+0x22/0x2f [ 4132.146375] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4132.149173] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4132.151879] ? watchdog+0x40/0x40 [ 4132.153430] ? hrtimer_interrupt+0x128/0x2c0 [ 4132.156030] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4132.158146] ? apic_timer_interrupt+0xf/0x20 [ 4132.160356] [ 4132.161726] ? native_safe_halt+0x2e/0x50 [ 4132.164043] kvm_wait+0xcd/0xe0 [ 4132.165904] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4132.168443] do_raw_spin_lock+0x177/0x190 [ 4132.170533] _raw_spin_lock+0x21/0x30 [ 4132.172399] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4132.174932] ? do_raw_spin_unlock+0x75/0x190 [ 4132.177366] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4132.180403] ? do_wait_intr+0xf0/0xf0 [ 4132.182289] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4132.185225] mdc_close+0x53b/0xd70 [mdc] [ 4132.187210] lmv_close+0x18d/0x490 [lmv] [ 4132.189449] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4132.192535] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4132.195396] ll_md_close+0x274/0x800 [lustre] [ 4132.198160] ll_file_release+0x128/0xc50 [lustre] [ 4132.201010] __fput+0x10d/0x440 [ 4132.203112] ____fput+0x12/0x20 [ 4132.204691] task_work_run+0xc6/0x110 [ 4132.206476] exit_to_usermode_loop+0x1dd/0x1f0 [ 4132.209069] do_syscall_64+0x3d6/0x440 [ 4132.211100] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4132.213272] RIP: 0033:0x7f1a63983cd8 [ 4132.215162] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4132.223708] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4132.227472] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4132.232163] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4132.236557] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4132.240934] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4132.245507] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4144.057712] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4144.060259] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4144.062600] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 4144.067494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4144.067566] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4144.067574] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4144.067576] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4144.067589] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4144.067593] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4144.067597] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4144.067599] RDX: 0000000000003110 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4144.067601] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4144.067603] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4144.067606] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4144.067609] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4144.067611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4144.067613] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4144.067623] Call Trace: [ 4144.067628] [ 4144.067960] ? show_regs.cold.9+0x22/0x2f [ 4144.067971] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4144.067978] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4144.067982] ? watchdog+0x40/0x40 [ 4144.067987] ? hrtimer_interrupt+0x128/0x2c0 [ 4144.067992] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4144.068000] ? apic_timer_interrupt+0xf/0x20 [ 4144.068005] [ 4144.068009] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4144.068013] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4144.068018] do_raw_spin_lock+0x177/0x190 [ 4144.068022] _raw_spin_lock+0x21/0x30 [ 4144.068027] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4144.068298] obd_mod_free+0x46/0x140 [ptlrpc] [ 4144.068680] mdc_commit_open+0x91/0xb0 [mdc] [ 4144.068712] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4144.069244] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4144.069513] after_reply+0xe2c/0x1780 [ptlrpc] [ 4144.072354] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4144.074220] ? __next_timer_interrupt+0x160/0x160 [ 4144.083827] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4144.083890] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4144.083897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4144.083901] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4144.083915] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4144.083919] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4144.083924] RAX: 000000000000711a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4144.083926] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4144.083928] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4144.083930] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4144.083932] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4144.083935] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4144.083938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4144.083939] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4144.083949] Call Trace: [ 4144.085037] [ 4144.085047] ? show_regs.cold.9+0x22/0x2f [ 4144.085061] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4144.085067] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4144.085072] ? watchdog+0x40/0x40 [ 4144.085077] ? hrtimer_interrupt+0x128/0x2c0 [ 4144.085081] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4144.085090] ? apic_timer_interrupt+0xf/0x20 [ 4144.085095] [ 4144.085098] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4144.085480] do_raw_spin_lock+0x177/0x190 [ 4144.085489] _raw_spin_lock+0x21/0x30 [ 4144.085495] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4144.141071] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4144.141402] ? do_wait_intr+0xf0/0xf0 [ 4144.141412] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4144.153225] ? do_raw_spin_unlock+0x75/0x190 [ 4144.153247] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4144.153831] ? do_wait_intr+0xf0/0xf0 [ 4144.153840] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4144.154047] mdc_close+0x53b/0xd70 [mdc] [ 4144.154086] lmv_close+0x18d/0x490 [lmv] [ 4144.166629] kthread+0x1d1/0x200 [ 4144.169326] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4144.169445] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4144.169525] ll_md_close+0x274/0x800 [lustre] [ 4144.169605] ll_file_release+0x128/0xc50 [lustre] [ 4144.169685] __fput+0x10d/0x440 [ 4144.169694] ____fput+0x12/0x20 [ 4144.169699] task_work_run+0xc6/0x110 [ 4144.169706] exit_to_usermode_loop+0x1dd/0x1f0 [ 4144.169713] do_syscall_64+0x3d6/0x440 [ 4144.169717] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4144.169726] RIP: 0033:0x7f1a63983cd8 [ 4144.169883] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4144.169887] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4144.169893] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4144.169895] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4144.169897] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4144.169899] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4144.169901] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4144.583062] ? set_kthread_struct+0x70/0x70 [ 4144.586485] ret_from_fork+0x1f/0x30 [ 4156.055204] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4156.059435] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4156.083736] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4156.095975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4156.101475] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4156.104686] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4156.112608] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4156.119211] RAX: 0000000000005a6b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4156.125367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4156.130818] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4156.135055] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4156.139322] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4156.145461] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4156.151196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4156.154667] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4156.160433] Call Trace: [ 4156.164210] [ 4156.165647] ? show_regs.cold.9+0x22/0x2f [ 4156.167387] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4156.170629] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4156.175438] ? watchdog+0x40/0x40 [ 4156.180933] ? hrtimer_interrupt+0x128/0x2c0 [ 4156.184790] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4156.189693] ? apic_timer_interrupt+0xf/0x20 [ 4156.202118] [ 4156.205142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4156.207339] do_raw_spin_lock+0x177/0x190 [ 4156.212096] _raw_spin_lock+0x21/0x30 [ 4156.218392] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4156.222758] ? do_raw_spin_unlock+0x75/0x190 [ 4156.228855] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4156.232999] ? do_wait_intr+0xf0/0xf0 [ 4156.235809] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4156.238803] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4156.241549] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4156.244912] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4156.247456] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4156.250388] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4156.253278] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4156.255370] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4156.259541] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4156.267426] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4156.274969] lmv_intent_open+0x338/0x1150 [lmv] [ 4156.282246] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4156.286455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4156.294598] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4156.302157] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4156.304819] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4156.308048] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4156.312419] ? do_raw_spin_unlock+0x75/0x190 [ 4156.315226] ? _raw_spin_unlock+0x12/0x30 [ 4156.318209] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4156.327757] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4156.330794] ll_atomic_open+0x36a/0xd70 [lustre] [ 4156.333133] lookup_open+0x356/0xb40 [ 4156.335471] do_last+0x313/0xfc0 [ 4156.337316] ? nd_jump_root+0xe5/0x160 [ 4156.338607] ? path_init+0x437/0x520 [ 4156.340858] path_openat+0xf7/0x500 [ 4156.342469] ? do_raw_spin_unlock+0x75/0x190 [ 4156.344502] do_filp_open+0x99/0x140 [ 4156.345683] ? getname_flags+0x6e/0x330 [ 4156.347591] ? __check_object_size+0xff/0x256 [ 4156.350056] ? do_raw_spin_unlock+0x75/0x190 [ 4156.360568] ? _raw_spin_unlock+0x12/0x30 [ 4156.366027] do_sys_openat2+0x2b4/0x410 [ 4156.367886] do_sys_open+0x73/0xa0 [ 4156.373445] __x64_sys_openat+0x24/0x30 [ 4156.378374] do_syscall_64+0xc1/0x440 [ 4156.383287] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4156.388932] RIP: 0033:0x7f1a63983332 [ 4156.390852] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4156.408299] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4156.411785] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4156.420530] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4156.432508] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4156.439109] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4156.441923] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4160.056116] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4160.061500] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4160.086268] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4160.091135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4160.096954] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4160.101083] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4160.109210] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4160.115208] RAX: 0000000000001e10 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4160.121713] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4160.128690] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4160.132727] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4160.136539] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4160.139177] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4160.142702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4160.145186] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4160.148309] Call Trace: [ 4160.149365] [ 4160.150410] ? show_regs.cold.9+0x22/0x2f [ 4160.151955] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4160.154090] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4160.156017] ? watchdog+0x40/0x40 [ 4160.157327] ? hrtimer_interrupt+0x128/0x2c0 [ 4160.159625] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4160.161784] ? apic_timer_interrupt+0xf/0x20 [ 4160.163479] [ 4160.164322] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4160.166519] do_raw_spin_lock+0x177/0x190 [ 4160.168330] _raw_spin_lock+0x21/0x30 [ 4160.169483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4160.171767] ? do_raw_spin_unlock+0x75/0x190 [ 4160.173105] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4160.175083] ? do_wait_intr+0xf0/0xf0 [ 4160.176457] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4160.179337] mdc_close+0x53b/0xd70 [mdc] [ 4160.181196] lmv_close+0x18d/0x490 [lmv] [ 4160.183035] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4160.185618] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4160.188158] ll_md_close+0x274/0x800 [lustre] [ 4160.190176] ll_file_release+0x128/0xc50 [lustre] [ 4160.196879] __fput+0x10d/0x440 [ 4160.198368] ____fput+0x12/0x20 [ 4160.200384] task_work_run+0xc6/0x110 [ 4160.202395] exit_to_usermode_loop+0x1dd/0x1f0 [ 4160.204525] do_syscall_64+0x3d6/0x440 [ 4160.206496] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4160.208210] RIP: 0033:0x7f1a63983cd8 [ 4160.209757] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4160.217217] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4160.220070] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4160.223190] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4160.226282] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4160.229090] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4160.232917] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4160.994552] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2137s! [ 4160.997656] Showing busy workqueues and worker pools: [ 4161.003771] workqueue events: flags=0x0 [ 4161.008453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4161.010600] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4161.017081] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4161.023936] pending: sec_gc_main [ptlrpc] [ 4161.028496] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4161.033796] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4161.048714] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4161.055237] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4161.062915] workqueue events_long: flags=0x0 [ 4161.066346] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4161.070312] pending: lu_objects_flush [obdclass] [ 4161.076790] workqueue events_unbound: flags=0x2 [ 4161.081209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4161.088084] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4161.097031] workqueue events_power_efficient: flags=0x80 [ 4161.099233] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4161.106173] pending: neigh_periodic_work, do_cache_clean [ 4161.107822] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4161.114109] pending: neigh_periodic_work, check_lifetime [ 4161.117093] workqueue mm_percpu_wq: flags=0x8 [ 4161.119240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4161.127793] pending: vmstat_update [ 4161.130356] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4161.135634] pending: vmstat_update [ 4161.137674] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4161.140817] pending: vmstat_update [ 4161.151143] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4161.158203] pending: vmstat_update [ 4161.170378] workqueue writeback: flags=0x4a [ 4161.173135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4161.179345] pending: wb_workfn, wb_workfn [ 4161.181216] workqueue xprtiod: flags=0xa [ 4161.183948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4161.189490] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4161.200794] workqueue ipv6_addrconf: flags=0x40008 [ 4161.202713] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4161.206312] pending: addrconf_verify_work [ 4161.209673] workqueue knbd0-recv: flags=0x1a [ 4161.213959] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4161.219445] in-flight: 647:recv_work [ 4161.223275] workqueue ptlrpc_pinger: flags=0x2 [ 4161.227163] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4161.233114] pending: ptlrpc_pinger_main [ptlrpc] [ 4161.238939] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4151s workers=2 idle: 652 [ 4172.057045] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4172.058509] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4172.060897] Modules linked in: lustre(O) [ 4172.067637] Modules linked in: [ 4172.067641] osp(O) ofd(O) [ 4172.070846] lustre(O) [ 4172.072336] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4172.072403] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4172.072411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4172.072413] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4172.072426] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4172.072430] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4172.072435] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4172.072437] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4172.072439] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4172.072441] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4172.072443] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4172.072446] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4172.072449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4172.072451] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4172.072459] Call Trace: [ 4172.072471] [ 4172.072476] ? show_regs.cold.9+0x22/0x2f [ 4172.072484] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4172.072491] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4172.072495] ? watchdog+0x40/0x40 [ 4172.072499] ? hrtimer_interrupt+0x128/0x2c0 [ 4172.072502] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4172.072508] ? apic_timer_interrupt+0xf/0x20 [ 4172.072512] [ 4172.072515] ? native_safe_halt+0x2e/0x50 [ 4172.072518] kvm_wait+0xcd/0xe0 [ 4172.072525] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4172.072532] do_raw_spin_lock+0x177/0x190 [ 4172.072536] _raw_spin_lock+0x21/0x30 [ 4172.072540] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4172.074906] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 4172.076453] ? do_raw_spin_unlock+0x75/0x190 [ 4172.076484] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4172.095746] lnet(O) [ 4172.102646] ? do_wait_intr+0xf0/0xf0 [ 4172.106205] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4172.106250] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4172.106258] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4172.106261] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4172.106274] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4172.106278] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4172.106282] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4172.106284] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4172.106286] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4172.106289] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4172.106290] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4172.106293] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4172.106296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4172.106299] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4172.106307] Call Trace: [ 4172.106311] [ 4172.106316] ? show_regs.cold.9+0x22/0x2f [ 4172.106325] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4172.106332] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4172.106336] ? watchdog+0x40/0x40 [ 4172.106340] ? hrtimer_interrupt+0x128/0x2c0 [ 4172.106344] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4172.106351] ? apic_timer_interrupt+0xf/0x20 [ 4172.106355] [ 4172.106358] ? native_safe_halt+0x2e/0x50 [ 4172.106362] kvm_wait+0xcd/0xe0 [ 4172.106369] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4172.106377] do_raw_spin_lock+0x177/0x190 [ 4172.106381] _raw_spin_lock+0x21/0x30 [ 4172.106386] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4172.109568] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4172.117523] obd_mod_free+0x46/0x140 [ptlrpc] [ 4172.124003] mdc_close+0x53b/0xd70 [mdc] [ 4172.126732] mdc_commit_open+0x91/0xb0 [mdc] [ 4172.129482] lmv_close+0x18d/0x490 [lmv] [ 4172.132281] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4172.132587] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4172.132922] after_reply+0xe2c/0x1780 [ptlrpc] [ 4172.136263] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4172.139353] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4172.143702] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4172.146066] ? __next_timer_interrupt+0x160/0x160 [ 4172.146089] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4172.146376] ? do_wait_intr+0xf0/0xf0 [ 4172.146386] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4172.146613] kthread+0x1d1/0x200 [ 4172.146620] ? set_kthread_struct+0x70/0x70 [ 4172.146623] ret_from_fork+0x1f/0x30 [ 4172.346316] ll_md_close+0x274/0x800 [lustre] [ 4172.348089] ll_file_release+0x128/0xc50 [lustre] [ 4172.351264] __fput+0x10d/0x440 [ 4172.352715] ____fput+0x12/0x20 [ 4172.354050] task_work_run+0xc6/0x110 [ 4172.355676] exit_to_usermode_loop+0x1dd/0x1f0 [ 4172.357824] do_syscall_64+0x3d6/0x440 [ 4172.359685] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4172.361827] RIP: 0033:0x7f1a63983cd8 [ 4172.363570] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4172.372253] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4172.375646] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4172.379809] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4172.384045] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4172.388666] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4172.392686] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4188.053027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4188.056661] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4188.058846] Modules linked in: lustre(O) [ 4188.062744] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4188.062902] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4188.062910] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4188.062913] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.063133] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4188.063142] RSP: 0018:ffffc38542073af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4188.063147] RAX: 00000000000073f6 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4188.063149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4188.063151] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4188.063153] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4188.063155] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4188.063161] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4188.063164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4188.063166] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4188.063175] Call Trace: [ 4188.063184] [ 4188.063191] ? show_regs.cold.9+0x22/0x2f [ 4188.063205] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4188.063212] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4188.063220] ? watchdog+0x40/0x40 [ 4188.063224] ? hrtimer_interrupt+0x128/0x2c0 [ 4188.063228] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4188.063240] ? apic_timer_interrupt+0xf/0x20 [ 4188.063245] [ 4188.063248] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.063255] do_raw_spin_lock+0x177/0x190 [ 4188.063264] _raw_spin_lock+0x21/0x30 [ 4188.063269] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4188.065726] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4188.065793] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4188.065800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4188.065802] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4188.065814] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4188.065818] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4188.065823] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4188.065825] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4188.065827] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4188.065829] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4188.065832] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4188.065834] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4188.065837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4188.065839] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4188.065847] Call Trace: [ 4188.065851] [ 4188.065856] ? show_regs.cold.9+0x22/0x2f [ 4188.065865] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4188.065871] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4188.065876] ? watchdog+0x40/0x40 [ 4188.065880] ? hrtimer_interrupt+0x128/0x2c0 [ 4188.065884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4188.066116] ? apic_timer_interrupt+0xf/0x20 [ 4188.066125] [ 4188.066129] ? native_safe_halt+0x2e/0x50 [ 4188.066133] kvm_wait+0xcd/0xe0 [ 4188.066141] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4188.066149] do_raw_spin_lock+0x177/0x190 [ 4188.066154] _raw_spin_lock+0x21/0x30 [ 4188.066158] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4188.066472] ? do_raw_spin_unlock+0x75/0x190 [ 4188.066478] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4188.066676] ? do_wait_intr+0xf0/0xf0 [ 4188.066685] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4188.066883] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4188.067398] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4188.067600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.067708] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4188.067751] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4188.067778] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4188.067806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.067887] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4188.068423] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4188.068455] lmv_intent_open+0x338/0x1150 [lmv] [ 4188.068488] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.068574] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.068655] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4188.068676] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4188.068758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.068842] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4188.069148] ? do_raw_spin_unlock+0x75/0x190 [ 4188.069155] ? _raw_spin_unlock+0x12/0x30 [ 4188.069162] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4188.069193] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4188.069213] ll_atomic_open+0x36a/0xd70 [lustre] [ 4188.069297] lookup_open+0x356/0xb40 [ 4188.102516] ? do_raw_spin_unlock+0x75/0x190 [ 4188.102539] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4188.103001] ? do_wait_intr+0xf0/0xf0 [ 4188.103017] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4188.103373] mdc_close+0x53b/0xd70 [mdc] [ 4188.103417] lmv_close+0x18d/0x490 [lmv] [ 4188.103619] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4188.103733] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4188.103818] ll_md_close+0x274/0x800 [lustre] [ 4188.103907] ll_file_release+0x128/0xc50 [lustre] [ 4188.104268] __fput+0x10d/0x440 [ 4188.104278] ____fput+0x12/0x20 [ 4188.104282] task_work_run+0xc6/0x110 [ 4188.104289] exit_to_usermode_loop+0x1dd/0x1f0 [ 4188.104296] do_syscall_64+0x3d6/0x440 [ 4188.104463] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4188.104474] RIP: 0033:0x7f1a63983cd8 [ 4188.104482] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4188.104485] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4188.104491] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4188.104492] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4188.104494] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4188.104496] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4188.104498] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4188.658550] do_last+0x313/0xfc0 [ 4188.661658] ? nd_jump_root+0xe5/0x160 [ 4188.662997] ? path_init+0x437/0x520 [ 4188.668887] path_openat+0xf7/0x500 [ 4188.670699] ? do_raw_spin_unlock+0x75/0x190 [ 4188.673210] do_filp_open+0x99/0x140 [ 4188.677952] ? getname_flags+0x6e/0x330 [ 4188.682474] ? __check_object_size+0xff/0x256 [ 4188.685239] ? do_raw_spin_unlock+0x75/0x190 [ 4188.694150] ? _raw_spin_unlock+0x12/0x30 [ 4188.699115] do_sys_openat2+0x2b4/0x410 [ 4188.700702] do_sys_open+0x73/0xa0 [ 4188.702432] __x64_sys_openat+0x24/0x30 [ 4188.708547] do_syscall_64+0xc1/0x440 [ 4188.710323] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4188.713575] RIP: 0033:0x7f1a63983332 [ 4188.716052] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4188.733729] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4188.742533] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4188.755692] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4188.772843] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4188.786620] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4188.797163] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4191.713057] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2168s! [ 4191.717760] Showing busy workqueues and worker pools: [ 4191.723702] workqueue events: flags=0x0 [ 4191.729146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4191.735451] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4191.743559] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.747561] pending: sec_gc_main [ptlrpc] [ 4191.750429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4191.754815] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4191.768270] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4191.770800] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4191.785020] workqueue events_long: flags=0x0 [ 4191.788359] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.791861] pending: lu_objects_flush [obdclass] [ 4191.796900] workqueue events_unbound: flags=0x2 [ 4191.802739] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4191.807664] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4191.821475] workqueue events_power_efficient: flags=0x80 [ 4191.827797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4191.837519] pending: neigh_periodic_work, do_cache_clean [ 4191.841974] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4191.847813] pending: neigh_periodic_work, check_lifetime [ 4191.853177] workqueue mm_percpu_wq: flags=0x8 [ 4191.856408] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.869091] pending: vmstat_update [ 4191.872805] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.883768] pending: vmstat_update [ 4191.897747] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.913879] pending: vmstat_update [ 4191.921926] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.934829] pending: vmstat_update [ 4191.939714] workqueue writeback: flags=0x4a [ 4191.945850] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4191.951516] pending: wb_workfn, wb_workfn [ 4191.954266] workqueue xprtiod: flags=0xa [ 4191.957886] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4191.962451] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4191.973302] workqueue ipv6_addrconf: flags=0x40008 [ 4191.975185] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4191.980766] pending: addrconf_verify_work [ 4191.982926] workqueue knbd0-recv: flags=0x1a [ 4191.986114] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4191.989725] in-flight: 647:recv_work [ 4191.991892] workqueue ptlrpc_pinger: flags=0x2 [ 4191.995218] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4191.999164] pending: ptlrpc_pinger_main [ptlrpc] [ 4192.004534] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4182s workers=2 idle: 652 [ 4200.057825] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4200.059757] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4200.062920] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4200.063004] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4200.063018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4200.063021] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4200.063042] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4200.063046] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4200.063051] RAX: 00000000000054e0 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4200.063054] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4200.063056] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4200.063058] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4200.063061] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4200.063064] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4200.063067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4200.063069] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4200.063080] Call Trace: [ 4200.063089] [ 4200.063095] ? show_regs.cold.9+0x22/0x2f [ 4200.063103] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4200.063110] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4200.063113] ? watchdog+0x40/0x40 [ 4200.063118] ? hrtimer_interrupt+0x128/0x2c0 [ 4200.063125] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4200.063134] ? apic_timer_interrupt+0xf/0x20 [ 4200.063138] [ 4200.063141] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4200.063150] do_raw_spin_lock+0x177/0x190 [ 4200.063155] _raw_spin_lock+0x21/0x30 [ 4200.063160] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4200.063686] ? do_raw_spin_unlock+0x75/0x190 [ 4200.063692] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4200.075339] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [ 4200.106205] ? do_wait_intr+0xf0/0xf0 [ 4200.106227] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4200.106840] mdc_close+0x53b/0xd70 [mdc] [ 4200.106885] lmv_close+0x18d/0x490 [lmv] [ 4200.106916] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4200.107092] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4200.107361] ll_md_close+0x274/0x800 [lustre] [ 4200.107517] ll_file_release+0x128/0xc50 [lustre] [ 4200.107598] __fput+0x10d/0x440 [ 4200.107607] ____fput+0x12/0x20 [ 4200.107612] task_work_run+0xc6/0x110 [ 4200.107620] exit_to_usermode_loop+0x1dd/0x1f0 [ 4200.107626] do_syscall_64+0x3d6/0x440 [ 4200.107631] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4200.107641] RIP: 0033:0x7f1a63983cd8 [ 4200.107648] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4200.107652] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4200.107657] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4200.107660] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4200.107662] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4200.107665] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4200.107667] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4200.440404] sha512_ssse3 sha512_generic [ 4200.442608] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4200.456011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4200.465360] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4200.467566] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4200.483312] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4200.488360] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4200.496131] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4200.502672] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4200.505833] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4200.511856] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4200.521005] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4200.525331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4200.532549] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4200.536984] Call Trace: [ 4200.539040] [ 4200.541454] ? show_regs.cold.9+0x22/0x2f [ 4200.544439] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4200.549342] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4200.551728] ? watchdog+0x40/0x40 [ 4200.553293] ? hrtimer_interrupt+0x128/0x2c0 [ 4200.555717] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4200.561398] ? apic_timer_interrupt+0xf/0x20 [ 4200.567423] [ 4200.569470] ? native_safe_halt+0x2e/0x50 [ 4200.573009] kvm_wait+0xcd/0xe0 [ 4200.574622] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4200.580616] do_raw_spin_lock+0x177/0x190 [ 4200.582852] _raw_spin_lock+0x21/0x30 [ 4200.586034] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4200.590126] obd_mod_free+0x46/0x140 [ptlrpc] [ 4200.594730] mdc_commit_open+0x91/0xb0 [mdc] [ 4200.598622] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4200.603452] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4200.608411] after_reply+0xe2c/0x1780 [ptlrpc] [ 4200.610283] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4200.612942] ? __next_timer_interrupt+0x160/0x160 [ 4200.617043] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4200.619343] ? do_wait_intr+0xf0/0xf0 [ 4200.622805] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4200.626792] kthread+0x1d1/0x200 [ 4200.628107] ? set_kthread_struct+0x70/0x70 [ 4200.632134] ret_from_fork+0x1f/0x30 [ 4216.053185] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4216.057192] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4216.058233] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4216.090268] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4216.090283] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4216.090286] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4216.090300] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4216.090304] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4216.090310] RAX: 0000000000003c1f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4216.090312] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4216.090315] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4216.090317] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4216.090319] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4216.090322] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4216.090325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4216.090328] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4216.090336] Call Trace: [ 4216.090341] [ 4216.090347] ? show_regs.cold.9+0x22/0x2f [ 4216.090356] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4216.090363] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4216.090367] ? watchdog+0x40/0x40 [ 4216.090371] ? hrtimer_interrupt+0x128/0x2c0 [ 4216.090374] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4216.090382] ? apic_timer_interrupt+0xf/0x20 [ 4216.090386] [ 4216.090389] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4216.090395] do_raw_spin_lock+0x177/0x190 [ 4216.090399] _raw_spin_lock+0x21/0x30 [ 4216.090402] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4216.093835] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4216.093938] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4216.093946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4216.093948] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4216.093962] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4216.093966] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4216.093971] RAX: 0000000000007e7a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4216.093973] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4216.093975] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4216.093977] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4216.093980] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4216.093983] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4216.093986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4216.093988] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4216.093996] Call Trace: [ 4216.094000] [ 4216.094011] ? show_regs.cold.9+0x22/0x2f [ 4216.094020] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4216.094027] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4216.094113] ? watchdog+0x40/0x40 [ 4216.094118] ? hrtimer_interrupt+0x128/0x2c0 [ 4216.094123] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4216.094138] ? apic_timer_interrupt+0xf/0x20 [ 4216.094142] [ 4216.094146] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4216.094156] do_raw_spin_lock+0x177/0x190 [ 4216.094160] _raw_spin_lock+0x21/0x30 [ 4216.094169] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4216.094550] ? do_raw_spin_unlock+0x75/0x190 [ 4216.094557] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4216.094972] ? do_wait_intr+0xf0/0xf0 [ 4216.094981] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4216.095308] mdc_close+0x53b/0xd70 [mdc] [ 4216.095421] lmv_close+0x18d/0x490 [lmv] [ 4216.095457] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4216.095579] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4216.103399] ? do_raw_spin_unlock+0x75/0x190 [ 4216.103418] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4216.103677] ? do_wait_intr+0xf0/0xf0 [ 4216.103685] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4216.103882] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4216.108428] ll_md_close+0x274/0x800 [lustre] [ 4216.108559] ll_file_release+0x128/0xc50 [lustre] [ 4216.108701] __fput+0x10d/0x440 [ 4216.108712] ____fput+0x12/0x20 [ 4216.108716] task_work_run+0xc6/0x110 [ 4216.108730] exit_to_usermode_loop+0x1dd/0x1f0 [ 4216.108739] do_syscall_64+0x3d6/0x440 [ 4216.108744] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4216.108753] RIP: 0033:0x7f1a63983cd8 [ 4216.108764] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4216.108769] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4216.108775] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4216.108777] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4216.108779] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4216.108782] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4216.108784] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4216.421436] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4216.424480] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.427247] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4216.429890] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4216.432165] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4216.434418] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.436966] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4216.439575] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4216.442302] lmv_intent_open+0x338/0x1150 [lmv] [ 4216.444873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.447430] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.449088] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4216.450748] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4216.452878] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.455403] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4216.458151] ? do_raw_spin_unlock+0x75/0x190 [ 4216.460232] ? _raw_spin_unlock+0x12/0x30 [ 4216.462521] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4216.464876] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4216.467453] ll_atomic_open+0x36a/0xd70 [lustre] [ 4216.470134] lookup_open+0x356/0xb40 [ 4216.471766] do_last+0x313/0xfc0 [ 4216.475942] ? nd_jump_root+0xe5/0x160 [ 4216.482052] ? path_init+0x437/0x520 [ 4216.483554] path_openat+0xf7/0x500 [ 4216.486328] ? do_raw_spin_unlock+0x75/0x190 [ 4216.488887] do_filp_open+0x99/0x140 [ 4216.491444] ? getname_flags+0x6e/0x330 [ 4216.493448] ? __check_object_size+0xff/0x256 [ 4216.495232] ? do_raw_spin_unlock+0x75/0x190 [ 4216.497300] ? _raw_spin_unlock+0x12/0x30 [ 4216.498902] do_sys_openat2+0x2b4/0x410 [ 4216.500852] do_sys_open+0x73/0xa0 [ 4216.502466] __x64_sys_openat+0x24/0x30 [ 4216.504236] do_syscall_64+0xc1/0x440 [ 4216.506204] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4216.508831] RIP: 0033:0x7f1a63983332 [ 4216.510581] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4216.519304] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4216.522912] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4216.526589] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4216.530248] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4216.534387] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4216.538209] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4222.440106] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2199s! [ 4222.448440] Showing busy workqueues and worker pools: [ 4222.453597] workqueue events: flags=0x0 [ 4222.457635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4222.466903] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4222.478300] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4222.484744] pending: sec_gc_main [ptlrpc] [ 4222.492112] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4222.499485] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4222.512472] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4222.527887] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4222.541924] workqueue events_long: flags=0x0 [ 4222.553693] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4222.562062] pending: lu_objects_flush [obdclass] [ 4222.575493] workqueue events_unbound: flags=0x2 [ 4222.577780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4222.587653] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4222.600240] workqueue events_power_efficient: flags=0x80 [ 4222.605499] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4222.613304] pending: neigh_periodic_work, do_cache_clean [ 4222.620074] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4222.633273] pending: neigh_periodic_work, check_lifetime [ 4222.640125] workqueue mm_percpu_wq: flags=0x8 [ 4222.641940] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4222.650628] pending: vmstat_update [ 4222.653313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4222.663729] pending: vmstat_update [ 4222.666886] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4222.676263] pending: vmstat_update [ 4222.680275] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4222.689887] pending: vmstat_update [ 4222.692461] workqueue writeback: flags=0x4a [ 4222.693669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4222.698165] pending: wb_workfn, wb_workfn [ 4222.708570] workqueue xprtiod: flags=0xa [ 4222.717370] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4222.726053] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4222.736570] workqueue ipv6_addrconf: flags=0x40008 [ 4222.743643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4222.745885] pending: addrconf_verify_work [ 4222.752160] workqueue knbd0-recv: flags=0x1a [ 4222.754364] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4222.762404] in-flight: 647:recv_work [ 4222.767900] workqueue ptlrpc_pinger: flags=0x2 [ 4222.770709] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4222.772441] pending: ptlrpc_pinger_main [ptlrpc] [ 4222.777690] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4212s workers=2 idle: 652 [ 4228.057098] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4228.058487] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4228.059551] Modules linked in: [ 4228.062381] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4228.062459] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4228.062467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4228.062470] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4228.062483] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4228.062487] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4228.062492] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4228.062494] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4228.062496] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4228.062498] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4228.062500] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4228.062503] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4228.062506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4228.062508] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4228.062517] Call Trace: [ 4228.062522] [ 4228.062528] ? show_regs.cold.9+0x22/0x2f [ 4228.062538] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4228.062545] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4228.062549] ? watchdog+0x40/0x40 [ 4228.062553] ? hrtimer_interrupt+0x128/0x2c0 [ 4228.062556] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4228.062563] ? apic_timer_interrupt+0xf/0x20 [ 4228.062567] [ 4228.062570] ? native_safe_halt+0x2e/0x50 [ 4228.062574] kvm_wait+0xcd/0xe0 [ 4228.062581] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4228.062588] do_raw_spin_lock+0x177/0x190 [ 4228.062592] _raw_spin_lock+0x21/0x30 [ 4228.062597] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4228.064240] lustre(O) [ 4228.089480] obd_mod_free+0x46/0x140 [ptlrpc] [ 4228.097779] osp(O) [ 4228.103599] mdc_commit_open+0x91/0xb0 [mdc] [ 4228.103649] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4228.103966] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4228.104213] after_reply+0xe2c/0x1780 [ptlrpc] [ 4228.104520] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4228.104726] ? __next_timer_interrupt+0x160/0x160 [ 4228.104737] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4228.105310] ? do_wait_intr+0xf0/0xf0 [ 4228.105330] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4228.105662] kthread+0x1d1/0x200 [ 4228.105668] ? set_kthread_struct+0x70/0x70 [ 4228.105672] ret_from_fork+0x1f/0x30 [ 4228.199609] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4228.213509] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4228.216529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4228.219399] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4228.221323] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4228.227079] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4228.229325] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4228.231887] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4228.233825] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4228.236117] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4228.238524] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4228.240882] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4228.243291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4228.246557] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4228.249228] Call Trace: [ 4228.249777] [ 4228.250938] ? show_regs.cold.9+0x22/0x2f [ 4228.251927] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4228.254006] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4228.255251] ? watchdog+0x40/0x40 [ 4228.256468] ? hrtimer_interrupt+0x128/0x2c0 [ 4228.258396] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4228.260175] ? apic_timer_interrupt+0xf/0x20 [ 4228.261532] [ 4228.262182] ? native_safe_halt+0x2e/0x50 [ 4228.263279] kvm_wait+0xcd/0xe0 [ 4228.264325] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4228.265958] do_raw_spin_lock+0x177/0x190 [ 4228.267801] _raw_spin_lock+0x21/0x30 [ 4228.268897] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4228.270606] ? do_raw_spin_unlock+0x75/0x190 [ 4228.272952] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4228.275178] ? do_wait_intr+0xf0/0xf0 [ 4228.277143] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4228.279702] mdc_close+0x53b/0xd70 [mdc] [ 4228.281665] lmv_close+0x18d/0x490 [lmv] [ 4228.283522] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4228.287287] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4228.290024] ll_md_close+0x274/0x800 [lustre] [ 4228.292756] ll_file_release+0x128/0xc50 [lustre] [ 4228.295333] __fput+0x10d/0x440 [ 4228.298678] ____fput+0x12/0x20 [ 4228.299772] task_work_run+0xc6/0x110 [ 4228.301331] exit_to_usermode_loop+0x1dd/0x1f0 [ 4228.303296] do_syscall_64+0x3d6/0x440 [ 4228.304638] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4228.307267] RIP: 0033:0x7f1a63983cd8 [ 4228.308511] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4228.317683] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4228.321326] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4228.325021] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4228.327993] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4228.330437] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4228.333266] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4242.459013] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4242.463582] rcu: 3-...!: (1335367 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 4242.474017] (detected by 1, t=2220053 jiffies, g=269633, q=901) [ 4242.477941] Sending NMI from CPU 1 to CPUs 3: [ 4242.481065] NMI backtrace for cpu 3 [ 4242.481071] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4242.481073] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4242.481074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4242.481076] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4242.481078] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 4242.481081] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4242.481082] RDX: 0000000000007dec RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4242.481083] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4242.481085] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4242.481086] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4242.481087] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4242.481088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4242.481089] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4242.481090] Call Trace: [ 4242.481091] [ 4242.481092] ? show_regs.cold.9+0x22/0x2f [ 4242.481093] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4242.481094] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4242.481095] ? nmi_handle+0xbd/0x210 [ 4242.481096] ? default_do_nmi+0x4b/0x1b0 [ 4242.481097] ? do_nmi+0x280/0x380 [ 4242.481098] ? end_repeat_nmi+0x16/0x1f [ 4242.481099] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4242.481100] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4242.481101] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4242.481102] [ 4242.481103] do_raw_spin_lock+0x177/0x190 [ 4242.481104] _raw_spin_lock+0x21/0x30 [ 4242.481105] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4242.481106] obd_mod_free+0x46/0x140 [ptlrpc] [ 4242.481107] mdc_commit_open+0x91/0xb0 [mdc] [ 4242.481108] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4242.481109] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4242.481110] after_reply+0xe2c/0x1780 [ptlrpc] [ 4242.481111] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4242.481112] ? __next_timer_interrupt+0x160/0x160 [ 4242.481113] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4242.481114] ? do_wait_intr+0xf0/0xf0 [ 4242.481115] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4242.481116] kthread+0x1d1/0x200 [ 4242.481117] ? set_kthread_struct+0x70/0x70 [ 4242.481118] ret_from_fork+0x1f/0x30 [ 4242.481169] rcu: rcu_sched kthread starved for 2219699 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 4242.621576] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4242.626511] rcu: RCU grace-period kthread stack dump: [ 4242.628982] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4242.634643] Call Trace: [ 4242.635866] __schedule+0x351/0xcb0 [ 4242.637574] schedule+0xc0/0x180 [ 4242.639136] schedule_timeout+0xb4/0x190 [ 4242.640893] ? do_raw_spin_unlock+0x75/0x190 [ 4242.642978] ? __next_timer_interrupt+0x160/0x160 [ 4242.645131] ? prepare_to_swait_event+0xf3/0x140 [ 4242.647189] rcu_gp_kthread+0x8dd/0x15b0 [ 4242.648998] ? force_qs_rnp+0x2c0/0x2c0 [ 4242.650740] kthread+0x1d1/0x200 [ 4242.652530] ? set_kthread_struct+0x70/0x70 [ 4242.654485] ret_from_fork+0x1f/0x30 [ 4242.656307] rcu: Stack dump where RCU GP kthread last ran: [ 4242.658974] Sending NMI from CPU 1 to CPUs 0: [ 4242.662659] NMI backtrace for cpu 0 [ 4242.662666] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4242.662668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4242.662670] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4242.662672] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4242.662743] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 [ 4242.662747] RAX: 000000000000077e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4242.662748] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4242.662750] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4242.662752] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4242.662753] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4242.662755] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4242.662756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4242.662757] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4242.662759] Call Trace: [ 4242.662759] [ 4242.662760] ? show_regs.cold.9+0x22/0x2f [ 4242.662762] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4242.662763] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4242.662764] ? nmi_handle+0xbd/0x210 [ 4242.662765] ? default_do_nmi+0x4b/0x1b0 [ 4242.662766] ? do_nmi+0x280/0x380 [ 4242.662767] ? end_repeat_nmi+0x16/0x1f [ 4242.662768] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4242.662769] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4242.662770] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4242.662771] [ 4242.662772] do_raw_spin_lock+0x177/0x190 [ 4242.662774] _raw_spin_lock+0x21/0x30 [ 4242.662775] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4242.662776] ? do_raw_spin_unlock+0x75/0x190 [ 4242.662785] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4242.662787] ? do_wait_intr+0xf0/0xf0 [ 4242.662788] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4242.662789] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4242.662791] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4242.662792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4242.662793] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4242.662794] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4242.662795] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4242.662796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4242.662798] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4242.662799] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4242.662800] lmv_intent_open+0x338/0x1150 [lmv] [ 4242.662801] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4242.662803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4242.662804] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4242.662805] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4242.662806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4242.662807] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4242.662808] ? do_raw_spin_unlock+0x75/0x190 [ 4242.662809] ? _raw_spin_unlock+0x12/0x30 [ 4242.662810] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4242.662811] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4242.662812] ll_atomic_open+0x36a/0xd70 [lustre] [ 4242.662813] lookup_open+0x356/0xb40 [ 4242.662814] do_last+0x313/0xfc0 [ 4242.662815] ? nd_jump_root+0xe5/0x160 [ 4242.662816] ? path_init+0x437/0x520 [ 4242.662817] path_openat+0xf7/0x500 [ 4242.662818] ? do_raw_spin_unlock+0x75/0x190 [ 4242.662819] do_filp_open+0x99/0x140 [ 4242.662820] ? getname_flags+0x6e/0x330 [ 4242.662821] ? __check_object_size+0xff/0x256 [ 4242.662822] ? do_raw_spin_unlock+0x75/0x190 [ 4242.662823] ? _raw_spin_unlock+0x12/0x30 [ 4242.662825] do_sys_openat2+0x2b4/0x410 [ 4242.662831] do_sys_open+0x73/0xa0 [ 4242.662832] __x64_sys_openat+0x24/0x30 [ 4242.662833] do_syscall_64+0xc1/0x440 [ 4242.662834] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4242.662835] RIP: 0033:0x7f1a63983332 [ 4242.662836] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4242.662838] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4242.662841] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4242.662842] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4242.662844] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4242.662845] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4242.662847] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4253.158724] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2230s! [ 4253.167602] Showing busy workqueues and worker pools: [ 4253.183000] workqueue events: flags=0x0 [ 4253.198127] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4253.214768] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4253.233945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4253.254996] pending: sec_gc_main [ptlrpc] [ 4253.275094] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4253.289774] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4253.304718] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4253.316908] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4253.341808] workqueue events_long: flags=0x0 [ 4253.348760] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4253.359081] pending: lu_objects_flush [obdclass] [ 4253.367584] workqueue events_unbound: flags=0x2 [ 4253.372622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4253.379772] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4253.397290] workqueue events_power_efficient: flags=0x80 [ 4253.414043] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4253.426343] pending: neigh_periodic_work, do_cache_clean [ 4253.430025] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4253.432531] pending: neigh_periodic_work, check_lifetime [ 4253.442295] workqueue mm_percpu_wq: flags=0x8 [ 4253.448322] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4253.456776] pending: vmstat_update [ 4253.460621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4253.472603] pending: vmstat_update [ 4253.476468] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4253.481689] pending: vmstat_update [ 4253.484504] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4253.493233] pending: vmstat_update [ 4253.496492] workqueue writeback: flags=0x4a [ 4253.499639] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4253.505646] pending: wb_workfn, wb_workfn [ 4253.512758] workqueue xprtiod: flags=0xa [ 4253.528384] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4253.540455] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4253.550910] workqueue ipv6_addrconf: flags=0x40008 [ 4253.574385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4253.590059] pending: addrconf_verify_work [ 4253.607557] workqueue knbd0-recv: flags=0x1a [ 4253.621600] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4253.636428] in-flight: 647:recv_work [ 4253.643771] workqueue ptlrpc_pinger: flags=0x2 [ 4253.653393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4253.658913] pending: ptlrpc_pinger_main [ptlrpc] [ 4253.662985] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4243s workers=2 idle: 652 [ 4256.059204] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4256.070769] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4256.094873] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4256.101118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4256.105181] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4256.108200] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4256.118194] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4256.122251] RAX: 0000000000007f40 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4256.125922] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4256.130154] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4256.134735] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4256.137941] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4256.140086] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4256.143507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4256.145588] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4256.151853] Call Trace: [ 4256.152816] [ 4256.153561] ? show_regs.cold.9+0x22/0x2f [ 4256.155141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4256.160193] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4256.162485] ? watchdog+0x40/0x40 [ 4256.163742] ? hrtimer_interrupt+0x128/0x2c0 [ 4256.164945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4256.166284] ? apic_timer_interrupt+0xf/0x20 [ 4256.167629] [ 4256.168302] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4256.170151] do_raw_spin_lock+0x177/0x190 [ 4256.171595] _raw_spin_lock+0x21/0x30 [ 4256.172912] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4256.175177] ? do_raw_spin_unlock+0x75/0x190 [ 4256.176967] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4256.179501] ? do_wait_intr+0xf0/0xf0 [ 4256.180743] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4256.183176] mdc_close+0x53b/0xd70 [mdc] [ 4256.184503] lmv_close+0x18d/0x490 [lmv] [ 4256.185939] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4256.188500] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4256.191504] ll_md_close+0x274/0x800 [lustre] [ 4256.194205] ll_file_release+0x128/0xc50 [lustre] [ 4256.196043] __fput+0x10d/0x440 [ 4256.197153] ____fput+0x12/0x20 [ 4256.198384] task_work_run+0xc6/0x110 [ 4256.200153] exit_to_usermode_loop+0x1dd/0x1f0 [ 4256.202605] do_syscall_64+0x3d6/0x440 [ 4256.204662] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4256.207024] RIP: 0033:0x7f1a63983cd8 [ 4256.208862] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4256.218593] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4256.222346] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4256.226148] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4256.230007] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4256.234118] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4256.237785] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4268.056171] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4268.058587] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4268.063608] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4268.063687] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4268.063695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4268.063697] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4268.063710] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4268.063714] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4268.063718] RAX: 0000000000006492 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4268.063720] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4268.063723] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4268.063725] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4268.063727] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4268.063730] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4268.063733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4268.063736] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4268.063744] Call Trace: [ 4268.063749] [ 4268.063755] ? show_regs.cold.9+0x22/0x2f [ 4268.063765] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4268.063771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4268.063775] ? watchdog+0x40/0x40 [ 4268.063780] ? hrtimer_interrupt+0x128/0x2c0 [ 4268.063784] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4268.063791] ? apic_timer_interrupt+0xf/0x20 [ 4268.063795] [ 4268.063798] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4268.063805] do_raw_spin_lock+0x177/0x190 [ 4268.063810] _raw_spin_lock+0x21/0x30 [ 4268.063815] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4268.068316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4268.068504] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4268.068511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4268.068514] RIP: 0010:__pv_queued_spin_lock_slowpath+0x185/0x6d0 [ 4268.068543] Code: 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 48 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 <0f> b6 03 84 c0 75 e6 48 83 05 2c d6 dc 02 01 44 89 f8 f0 66 44 0f [ 4268.068547] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4268.068552] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4268.068558] RDX: 0000000000002cd2 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4268.068561] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4268.068563] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4268.068566] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4268.068569] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4268.068572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4268.068574] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4268.068582] Call Trace: [ 4268.068587] [ 4268.068592] ? show_regs.cold.9+0x22/0x2f [ 4268.068663] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4268.068670] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4268.068674] ? watchdog+0x40/0x40 [ 4268.068685] ? hrtimer_interrupt+0x128/0x2c0 [ 4268.068688] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4268.068696] ? apic_timer_interrupt+0xf/0x20 [ 4268.068701] [ 4268.068703] ? __pv_queued_spin_lock_slowpath+0x185/0x6d0 [ 4268.068707] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4268.068712] do_raw_spin_lock+0x177/0x190 [ 4268.068716] _raw_spin_lock+0x21/0x30 [ 4268.068725] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4268.069019] obd_mod_free+0x46/0x140 [ptlrpc] [ 4268.069442] mdc_commit_open+0x91/0xb0 [mdc] [ 4268.069481] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4268.069730] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4268.070148] after_reply+0xe2c/0x1780 [ptlrpc] [ 4268.070371] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4268.070854] ? __next_timer_interrupt+0x160/0x160 [ 4268.070869] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4268.071199] ? do_wait_intr+0xf0/0xf0 [ 4268.071208] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4268.116906] ? do_raw_spin_unlock+0x75/0x190 [ 4268.116926] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4268.117205] ? do_wait_intr+0xf0/0xf0 [ 4268.117213] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4268.117514] mdc_close+0x53b/0xd70 [mdc] [ 4268.117626] lmv_close+0x18d/0x490 [lmv] [ 4268.117657] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4268.117773] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4268.117854] ll_md_close+0x274/0x800 [lustre] [ 4268.117936] ll_file_release+0x128/0xc50 [lustre] [ 4268.118020] __fput+0x10d/0x440 [ 4268.118030] ____fput+0x12/0x20 [ 4268.118034] task_work_run+0xc6/0x110 [ 4268.118041] exit_to_usermode_loop+0x1dd/0x1f0 [ 4268.118047] do_syscall_64+0x3d6/0x440 [ 4268.118051] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4268.118059] RIP: 0033:0x7f1a63983cd8 [ 4268.118067] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4268.118070] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4268.118075] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4268.118077] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4268.118080] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4268.118082] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4268.118084] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4268.541802] kthread+0x1d1/0x200 [ 4268.543208] ? set_kthread_struct+0x70/0x70 [ 4268.545099] ret_from_fork+0x1f/0x30 [ 4280.057553] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 4280.062065] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4280.162373] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4280.182103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4280.201568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4280.210305] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4280.231743] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4280.236765] RAX: 0000000000005083 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4280.253821] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4280.273115] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4280.280249] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4280.290647] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4280.301211] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4280.316682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4280.329504] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4280.359114] Call Trace: [ 4280.366022] [ 4280.366794] ? show_regs.cold.9+0x22/0x2f [ 4280.375595] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4280.378320] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4280.390027] ? watchdog+0x40/0x40 [ 4280.393111] ? hrtimer_interrupt+0x128/0x2c0 [ 4280.394736] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4280.399951] ? apic_timer_interrupt+0xf/0x20 [ 4280.409078] [ 4280.410172] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4280.415968] do_raw_spin_lock+0x177/0x190 [ 4280.419105] _raw_spin_lock+0x21/0x30 [ 4280.421567] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4280.428645] ? do_raw_spin_unlock+0x75/0x190 [ 4280.439002] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4280.444502] ? do_wait_intr+0xf0/0xf0 [ 4280.448153] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4280.459958] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4280.466563] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4280.472100] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4280.477363] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4280.484635] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4280.486243] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4280.493353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4280.496272] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4280.504016] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4280.514085] lmv_intent_open+0x338/0x1150 [lmv] [ 4280.525313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4280.534099] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4280.542706] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4280.546553] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4280.548336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4280.558115] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4280.562997] ? do_raw_spin_unlock+0x75/0x190 [ 4280.570173] ? _raw_spin_unlock+0x12/0x30 [ 4280.573561] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4280.582309] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4280.584252] ll_atomic_open+0x36a/0xd70 [lustre] [ 4280.589575] lookup_open+0x356/0xb40 [ 4280.598648] do_last+0x313/0xfc0 [ 4280.601235] ? nd_jump_root+0xe5/0x160 [ 4280.605540] ? path_init+0x437/0x520 [ 4280.608693] path_openat+0xf7/0x500 [ 4280.609972] ? do_raw_spin_unlock+0x75/0x190 [ 4280.617032] do_filp_open+0x99/0x140 [ 4280.619088] ? getname_flags+0x6e/0x330 [ 4280.625131] ? __check_object_size+0xff/0x256 [ 4280.628045] ? do_raw_spin_unlock+0x75/0x190 [ 4280.633248] ? _raw_spin_unlock+0x12/0x30 [ 4280.635787] do_sys_openat2+0x2b4/0x410 [ 4280.639632] do_sys_open+0x73/0xa0 [ 4280.642713] __x64_sys_openat+0x24/0x30 [ 4280.649859] do_syscall_64+0xc1/0x440 [ 4280.653445] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4280.658697] RIP: 0033:0x7f1a63983332 [ 4280.660858] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4280.686209] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4280.690388] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4280.708704] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4280.721671] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4280.726971] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4280.731272] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4283.873669] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2260s! [ 4283.887961] Showing busy workqueues and worker pools: [ 4283.894621] workqueue events: flags=0x0 [ 4283.900822] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4283.913150] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4283.922352] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4283.927781] pending: sec_gc_main [ptlrpc] [ 4283.930796] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4283.940110] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4283.945794] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4283.950015] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4283.958787] workqueue events_long: flags=0x0 [ 4283.960633] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4283.969263] pending: lu_objects_flush [obdclass] [ 4283.972767] workqueue events_unbound: flags=0x2 [ 4283.974723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4283.979253] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4283.987553] workqueue events_power_efficient: flags=0x80 [ 4283.990459] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4283.993882] pending: neigh_periodic_work, do_cache_clean [ 4283.997266] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4284.002713] pending: neigh_periodic_work, check_lifetime [ 4284.005213] workqueue mm_percpu_wq: flags=0x8 [ 4284.008339] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4284.011253] pending: vmstat_update [ 4284.012592] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4284.017145] pending: vmstat_update [ 4284.019450] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4284.026999] pending: vmstat_update [ 4284.030565] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4284.036112] pending: vmstat_update [ 4284.039259] workqueue writeback: flags=0x4a [ 4284.042624] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4284.054368] pending: wb_workfn, wb_workfn [ 4284.057027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4284.060068] workqueue xprtiod: flags=0xa [ 4284.063115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4284.063171] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4284.063176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4284.063179] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4284.063190] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4284.063193] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4284.063196] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4284.063199] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4284.063201] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4284.063203] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4284.063205] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4284.063266] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4284.063269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4284.063270] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4284.063277] Call Trace: [ 4284.063282] [ 4284.063286] ? show_regs.cold.9+0x22/0x2f [ 4284.063294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4284.063299] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4284.063302] ? watchdog+0x40/0x40 [ 4284.063305] ? hrtimer_interrupt+0x128/0x2c0 [ 4284.063307] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4284.063312] ? apic_timer_interrupt+0xf/0x20 [ 4284.063316] [ 4284.063317] ? native_safe_halt+0x2e/0x50 [ 4284.063320] kvm_wait+0xcd/0xe0 [ 4284.063325] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4284.063388] do_raw_spin_lock+0x177/0x190 [ 4284.063394] _raw_spin_lock+0x21/0x30 [ 4284.063397] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4284.063774] ? do_raw_spin_unlock+0x75/0x190 [ 4284.063781] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4284.063975] ? do_wait_intr+0xf0/0xf0 [ 4284.063983] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4284.064173] mdc_close+0x53b/0xd70 [mdc] [ 4284.064271] lmv_close+0x18d/0x490 [lmv] [ 4284.064299] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4284.064693] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4284.064776] ll_md_close+0x274/0x800 [lustre] [ 4284.064830] ll_file_release+0x128/0xc50 [lustre] [ 4284.064882] __fput+0x10d/0x440 [ 4284.064889] ____fput+0x12/0x20 [ 4284.064891] task_work_run+0xc6/0x110 [ 4284.064896] exit_to_usermode_loop+0x1dd/0x1f0 [ 4284.064901] do_syscall_64+0x3d6/0x440 [ 4284.064904] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4284.064909] RIP: 0033:0x7f1a63983cd8 [ 4284.064915] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4284.064917] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4284.064921] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4284.064922] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4284.064924] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4284.064925] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4284.064927] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4284.380310] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4284.392961] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4284.404809] workqueue ipv6_addrconf: flags=0x40008 [ 4284.407776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4284.415488] pending: addrconf_verify_work [ 4284.417106] workqueue knbd0-recv: flags=0x1a [ 4284.420890] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4284.430110] in-flight: 647:recv_work [ 4284.433641] workqueue ptlrpc_pinger: flags=0x2 [ 4284.438784] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4284.447899] pending: ptlrpc_pinger_main [ptlrpc] [ 4284.451178] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4274s workers=2 idle: 652 [ 4296.064814] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4296.068560] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4296.078885] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4296.078973] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4296.078980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4296.078983] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4296.078997] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4296.079002] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4296.079006] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4296.079008] RDX: 0000000000002c2a RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4296.079010] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4296.079012] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4296.079014] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4296.079017] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4296.079020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4296.079023] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4296.079031] Call Trace: [ 4296.079036] [ 4296.079042] ? show_regs.cold.9+0x22/0x2f [ 4296.079050] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4296.079058] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4296.079063] ? watchdog+0x40/0x40 [ 4296.079067] ? hrtimer_interrupt+0x128/0x2c0 [ 4296.079071] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4296.079080] ? apic_timer_interrupt+0xf/0x20 [ 4296.079085] [ 4296.079088] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4296.079093] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4296.079097] do_raw_spin_lock+0x177/0x190 [ 4296.079103] _raw_spin_lock+0x21/0x30 [ 4296.079108] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4296.087146] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4296.087228] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4296.087235] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4296.087238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4296.087251] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4296.089297] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4296.089306] RAX: 0000000000000e86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4296.089309] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4296.089311] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4296.089313] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4296.089315] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4296.089318] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4296.089321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4296.089323] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4296.089331] Call Trace: [ 4296.089341] [ 4296.089347] ? show_regs.cold.9+0x22/0x2f [ 4296.089359] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4296.089366] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4296.089374] ? watchdog+0x40/0x40 [ 4296.089378] ? hrtimer_interrupt+0x128/0x2c0 [ 4296.089383] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4296.089391] ? apic_timer_interrupt+0xf/0x20 [ 4296.089396] [ 4296.089399] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4296.089407] do_raw_spin_lock+0x177/0x190 [ 4296.089412] _raw_spin_lock+0x21/0x30 [ 4296.089418] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4296.162307] obd_mod_free+0x46/0x140 [ptlrpc] [ 4296.162756] mdc_commit_open+0x91/0xb0 [mdc] [ 4296.162796] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4296.163118] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4296.163323] after_reply+0xe2c/0x1780 [ptlrpc] [ 4296.164040] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4296.178970] ? do_raw_spin_unlock+0x75/0x190 [ 4296.178996] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4296.181969] ? do_wait_intr+0xf0/0xf0 [ 4296.182049] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4296.182559] mdc_close+0x53b/0xd70 [mdc] [ 4296.182612] lmv_close+0x18d/0x490 [lmv] [ 4296.182648] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4296.182791] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4296.182891] ll_md_close+0x274/0x800 [lustre] [ 4296.182989] ll_file_release+0x128/0xc50 [lustre] [ 4296.183091] __fput+0x10d/0x440 [ 4296.183100] ____fput+0x12/0x20 [ 4296.183104] task_work_run+0xc6/0x110 [ 4296.183111] exit_to_usermode_loop+0x1dd/0x1f0 [ 4296.183119] do_syscall_64+0x3d6/0x440 [ 4296.183123] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4296.183132] RIP: 0033:0x7f1a63983cd8 [ 4296.183140] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4296.183144] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4296.183149] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4296.183152] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4296.183154] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4296.183156] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4296.183158] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4296.579293] ? __next_timer_interrupt+0x160/0x160 [ 4296.583511] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4296.585932] ? do_wait_intr+0xf0/0xf0 [ 4296.587878] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4296.590533] kthread+0x1d1/0x200 [ 4296.592002] ? set_kthread_struct+0x70/0x70 [ 4296.593959] ret_from_fork+0x1f/0x30 [ 4312.057826] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4312.061406] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4312.067849] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4312.067965] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4312.067973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4312.067976] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.067990] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4312.067995] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4312.068004] RAX: 0000000000001785 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4312.068006] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4312.068009] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4312.068011] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4312.068014] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4312.068021] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4312.068024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4312.068027] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4312.068035] Call Trace: [ 4312.068221] [ 4312.068229] ? show_regs.cold.9+0x22/0x2f [ 4312.068239] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4312.068251] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4312.068255] ? watchdog+0x40/0x40 [ 4312.068260] ? hrtimer_interrupt+0x128/0x2c0 [ 4312.068264] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4312.068273] ? apic_timer_interrupt+0xf/0x20 [ 4312.068283] [ 4312.068286] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.068293] do_raw_spin_lock+0x177/0x190 [ 4312.068298] _raw_spin_lock+0x21/0x30 [ 4312.068307] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4312.068705] ? do_raw_spin_unlock+0x75/0x190 [ 4312.068711] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4312.069443] ? do_wait_intr+0xf0/0xf0 [ 4312.069452] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4312.069761] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4312.070306] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4312.070775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4312.086583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4312.086664] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4312.086670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4312.086674] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.086689] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4312.086694] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4312.086698] RAX: 0000000000002490 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4312.086700] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4312.086702] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4312.086704] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4312.086707] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4312.086710] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4312.086724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4312.086727] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4312.086735] Call Trace: [ 4312.086740] [ 4312.086746] ? show_regs.cold.9+0x22/0x2f [ 4312.086754] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4312.086761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4312.086765] ? watchdog+0x40/0x40 [ 4312.086769] ? hrtimer_interrupt+0x128/0x2c0 [ 4312.086773] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4312.086783] ? apic_timer_interrupt+0xf/0x20 [ 4312.086788] [ 4312.086791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.086797] do_raw_spin_lock+0x177/0x190 [ 4312.086802] _raw_spin_lock+0x21/0x30 [ 4312.086807] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4312.087083] ? do_raw_spin_unlock+0x75/0x190 [ 4312.087089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4312.087294] ? do_wait_intr+0xf0/0xf0 [ 4312.087302] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4312.087491] mdc_close+0x53b/0xd70 [mdc] [ 4312.087528] lmv_close+0x18d/0x490 [lmv] [ 4312.087558] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4312.087667] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4312.087752] ll_md_close+0x274/0x800 [lustre] [ 4312.087837] ll_file_release+0x128/0xc50 [lustre] [ 4312.087924] __fput+0x10d/0x440 [ 4312.087932] ____fput+0x12/0x20 [ 4312.087937] task_work_run+0xc6/0x110 [ 4312.087943] exit_to_usermode_loop+0x1dd/0x1f0 [ 4312.087950] do_syscall_64+0x3d6/0x440 [ 4312.087955] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4312.087962] RIP: 0033:0x7f1a63983cd8 [ 4312.087968] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4312.087972] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4312.087977] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4312.087980] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4312.087982] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4312.087984] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4312.087986] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4312.799987] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4312.814734] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4312.816063] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4312.828055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4312.833787] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4312.837256] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4312.839914] lmv_intent_open+0x338/0x1150 [lmv] [ 4312.854892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4312.869266] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4312.879248] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4312.882902] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4312.888560] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4312.892998] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4312.897149] ? do_raw_spin_unlock+0x75/0x190 [ 4312.900169] ? _raw_spin_unlock+0x12/0x30 [ 4312.906909] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4312.909617] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4312.917566] ll_atomic_open+0x36a/0xd70 [lustre] [ 4312.923669] lookup_open+0x356/0xb40 [ 4312.925643] do_last+0x313/0xfc0 [ 4312.929655] ? nd_jump_root+0xe5/0x160 [ 4312.933962] ? path_init+0x437/0x520 [ 4312.938234] path_openat+0xf7/0x500 [ 4312.943603] ? do_raw_spin_unlock+0x75/0x190 [ 4312.952747] do_filp_open+0x99/0x140 [ 4312.954263] ? getname_flags+0x6e/0x330 [ 4312.957493] ? __check_object_size+0xff/0x256 [ 4312.964763] ? do_raw_spin_unlock+0x75/0x190 [ 4312.969299] ? _raw_spin_unlock+0x12/0x30 [ 4312.974246] do_sys_openat2+0x2b4/0x410 [ 4312.976772] do_sys_open+0x73/0xa0 [ 4312.982300] __x64_sys_openat+0x24/0x30 [ 4312.988124] do_syscall_64+0xc1/0x440 [ 4312.996955] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4313.005792] RIP: 0033:0x7f1a63983332 [ 4313.010307] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4313.045657] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4313.060640] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4313.065623] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4313.093989] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4313.109867] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4313.120419] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4314.602538] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2291s! [ 4314.611908] Showing busy workqueues and worker pools: [ 4314.620038] workqueue events: flags=0x0 [ 4314.622809] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4314.635523] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4314.644774] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4314.652652] pending: sec_gc_main [ptlrpc] [ 4314.666934] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4314.674989] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4314.689299] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4314.695282] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4314.701644] workqueue events_long: flags=0x0 [ 4314.705198] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4314.717683] pending: lu_objects_flush [obdclass] [ 4314.725549] workqueue events_unbound: flags=0x2 [ 4314.730402] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4314.741934] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4314.758156] workqueue events_power_efficient: flags=0x80 [ 4314.768937] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4314.778773] pending: neigh_periodic_work, do_cache_clean [ 4314.797861] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4314.830010] pending: neigh_periodic_work, check_lifetime [ 4314.843908] workqueue mm_percpu_wq: flags=0x8 [ 4314.853162] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4314.869181] pending: vmstat_update [ 4314.871836] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4314.887418] pending: vmstat_update [ 4314.900325] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4314.914660] pending: vmstat_update [ 4314.918289] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4314.926950] pending: vmstat_update [ 4314.937213] workqueue writeback: flags=0x4a [ 4314.942212] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4314.951731] pending: wb_workfn, wb_workfn [ 4314.963289] workqueue xprtiod: flags=0xa [ 4314.969137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4314.975195] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4314.989366] workqueue ipv6_addrconf: flags=0x40008 [ 4314.994273] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4315.004551] pending: addrconf_verify_work [ 4315.008422] workqueue knbd0-recv: flags=0x1a [ 4315.012570] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4315.017824] in-flight: 647:recv_work [ 4315.021209] workqueue ptlrpc_pinger: flags=0x2 [ 4315.024134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4315.028583] pending: ptlrpc_pinger_main [ptlrpc] [ 4315.034802] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4305s workers=2 idle: 652 [ 4324.056021] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4324.058146] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4324.062620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4324.062699] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4324.062706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4324.062710] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4324.062722] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4324.062732] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4324.062737] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4324.062739] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4324.062741] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4324.062743] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4324.062745] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4324.062749] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4324.062751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4324.062754] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4324.062762] Call Trace: [ 4324.062768] [ 4324.062772] ? show_regs.cold.9+0x22/0x2f [ 4324.062782] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4324.062788] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4324.062792] ? watchdog+0x40/0x40 [ 4324.062796] ? hrtimer_interrupt+0x128/0x2c0 [ 4324.062799] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4324.062806] ? apic_timer_interrupt+0xf/0x20 [ 4324.062811] [ 4324.062813] ? native_safe_halt+0x2e/0x50 [ 4324.062834] kvm_wait+0xcd/0xe0 [ 4324.062849] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4324.062933] do_raw_spin_lock+0x177/0x190 [ 4324.062939] _raw_spin_lock+0x21/0x30 [ 4324.062942] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4324.063441] ? do_raw_spin_unlock+0x75/0x190 [ 4324.063450] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4324.063755] ? do_wait_intr+0xf0/0xf0 [ 4324.063764] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4324.070588] Modules linked in: lustre(O) [ 4324.095617] mdc_close+0x53b/0xd70 [mdc] [ 4324.095666] lmv_close+0x18d/0x490 [lmv] [ 4324.095694] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4324.095802] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4324.096242] ll_md_close+0x274/0x800 [lustre] [ 4324.109869] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4324.109943] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4324.109951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4324.109954] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4324.109969] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4324.109973] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4324.109996] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4324.109998] RDX: 0000000000000ad2 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4324.110000] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4324.110002] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4324.110005] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4324.110008] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4324.110011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4324.110013] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4324.110022] Call Trace: [ 4324.110028] [ 4324.110034] ? show_regs.cold.9+0x22/0x2f [ 4324.110045] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4324.110052] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4324.110057] ? watchdog+0x40/0x40 [ 4324.110062] ? hrtimer_interrupt+0x128/0x2c0 [ 4324.110066] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4324.110076] ? apic_timer_interrupt+0xf/0x20 [ 4324.110080] [ 4324.110083] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4324.110088] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4324.110092] do_raw_spin_lock+0x177/0x190 [ 4324.110097] _raw_spin_lock+0x21/0x30 [ 4324.110103] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4324.114700] ll_file_release+0x128/0xc50 [lustre] [ 4324.119884] obd_mod_free+0x46/0x140 [ptlrpc] [ 4324.149929] __fput+0x10d/0x440 [ 4324.149951] ____fput+0x12/0x20 [ 4324.149957] task_work_run+0xc6/0x110 [ 4324.149966] exit_to_usermode_loop+0x1dd/0x1f0 [ 4324.149973] do_syscall_64+0x3d6/0x440 [ 4324.150249] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4324.150258] RIP: 0033:0x7f1a63983cd8 [ 4324.150267] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4324.150272] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4324.150277] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4324.150279] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4324.150281] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4324.150283] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4324.150286] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4324.737566] mdc_commit_open+0x91/0xb0 [mdc] [ 4324.740343] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4324.743245] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4324.746098] after_reply+0xe2c/0x1780 [ptlrpc] [ 4324.748445] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4324.751148] ? __next_timer_interrupt+0x160/0x160 [ 4324.753357] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4324.757753] ? do_wait_intr+0xf0/0xf0 [ 4324.762731] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4324.769129] kthread+0x1d1/0x200 [ 4324.771367] ? set_kthread_struct+0x70/0x70 [ 4324.774237] ret_from_fork+0x1f/0x30 [ 4340.055091] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4340.057158] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4340.068356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.068446] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.068453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4340.068456] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.068470] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4340.068475] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4340.068480] RAX: 0000000000007f11 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4340.068482] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4340.068485] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4340.068487] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4340.068489] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4340.068493] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4340.068496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4340.068498] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4340.068507] Call Trace: [ 4340.068513] [ 4340.068519] ? show_regs.cold.9+0x22/0x2f [ 4340.068529] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4340.068535] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4340.068539] ? watchdog+0x40/0x40 [ 4340.068543] ? hrtimer_interrupt+0x128/0x2c0 [ 4340.068547] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4340.068554] ? apic_timer_interrupt+0xf/0x20 [ 4340.068559] [ 4340.068562] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.068567] do_raw_spin_lock+0x177/0x190 [ 4340.068731] _raw_spin_lock+0x21/0x30 [ 4340.068739] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4340.070696] ? do_raw_spin_unlock+0x75/0x190 [ 4340.070705] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4340.073376] ? do_wait_intr+0xf0/0xf0 [ 4340.073390] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4340.073627] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4340.073864] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4340.074088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4340.074405] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4340.074455] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4340.074484] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4340.074511] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4340.074607] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4340.075411] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4340.075453] lmv_intent_open+0x338/0x1150 [lmv] [ 4340.075483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4340.075574] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4340.082291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.082376] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.082383] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4340.082386] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4340.082399] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4340.082403] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4340.082407] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4340.082410] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4340.082412] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4340.082414] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4340.082417] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4340.082421] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4340.082424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4340.082426] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4340.082439] Call Trace: [ 4340.082449] [ 4340.082558] ? show_regs.cold.9+0x22/0x2f [ 4340.083222] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4340.083230] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4340.083239] ? watchdog+0x40/0x40 [ 4340.083243] ? hrtimer_interrupt+0x128/0x2c0 [ 4340.083247] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4340.083256] ? apic_timer_interrupt+0xf/0x20 [ 4340.083260] [ 4340.083263] ? native_safe_halt+0x2e/0x50 [ 4340.083268] kvm_wait+0xcd/0xe0 [ 4340.083275] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4340.083283] do_raw_spin_lock+0x177/0x190 [ 4340.083293] _raw_spin_lock+0x21/0x30 [ 4340.083301] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4340.157077] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4340.157120] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4340.157234] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4340.157303] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4340.157382] ? do_raw_spin_unlock+0x75/0x190 [ 4340.157392] ? _raw_spin_unlock+0x12/0x30 [ 4340.157398] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4340.157424] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4340.157444] ll_atomic_open+0x36a/0xd70 [lustre] [ 4340.157535] lookup_open+0x356/0xb40 [ 4340.157545] do_last+0x313/0xfc0 [ 4340.157552] ? nd_jump_root+0xe5/0x160 [ 4340.157559] ? path_init+0x437/0x520 [ 4340.157563] path_openat+0xf7/0x500 [ 4340.157570] ? do_raw_spin_unlock+0x75/0x190 [ 4340.157575] do_filp_open+0x99/0x140 [ 4340.157583] ? getname_flags+0x6e/0x330 [ 4340.157590] ? __check_object_size+0xff/0x256 [ 4340.157596] ? do_raw_spin_unlock+0x75/0x190 [ 4340.157600] ? _raw_spin_unlock+0x12/0x30 [ 4340.157606] do_sys_openat2+0x2b4/0x410 [ 4340.157611] do_sys_open+0x73/0xa0 [ 4340.157615] __x64_sys_openat+0x24/0x30 [ 4340.157618] do_syscall_64+0xc1/0x440 [ 4340.157624] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4340.157630] RIP: 0033:0x7f1a63983332 [ 4340.157639] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4340.157643] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4340.157648] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4340.157650] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4340.157652] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4340.157654] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4340.157656] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4340.748273] ? do_raw_spin_unlock+0x75/0x190 [ 4340.756456] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4340.758478] ? do_wait_intr+0xf0/0xf0 [ 4340.764737] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4340.767676] mdc_close+0x53b/0xd70 [mdc] [ 4340.773601] lmv_close+0x18d/0x490 [lmv] [ 4340.775813] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4340.781576] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4340.784723] ll_md_close+0x274/0x800 [lustre] [ 4340.786505] ll_file_release+0x128/0xc50 [lustre] [ 4340.798096] __fput+0x10d/0x440 [ 4340.799287] ____fput+0x12/0x20 [ 4340.800431] task_work_run+0xc6/0x110 [ 4340.805459] exit_to_usermode_loop+0x1dd/0x1f0 [ 4340.809238] do_syscall_64+0x3d6/0x440 [ 4340.813569] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4340.819123] RIP: 0033:0x7f1a63983cd8 [ 4340.820782] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4340.832237] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4340.836707] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4340.840423] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4340.843692] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4340.847328] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4340.850568] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4345.318202] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2322s! [ 4345.334269] Showing busy workqueues and worker pools: [ 4345.338078] workqueue events: flags=0x0 [ 4345.342015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4345.351660] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4345.359100] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4345.362975] pending: sec_gc_main [ptlrpc] [ 4345.370711] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4345.376925] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4345.385411] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4345.392757] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4345.401167] workqueue events_long: flags=0x0 [ 4345.403335] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4345.412129] pending: lu_objects_flush [obdclass] [ 4345.415120] workqueue events_unbound: flags=0x2 [ 4345.418034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4345.425803] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4345.439837] workqueue events_power_efficient: flags=0x80 [ 4345.451181] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4345.456834] pending: neigh_periodic_work, do_cache_clean [ 4345.460733] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4345.465856] pending: neigh_periodic_work, check_lifetime [ 4345.468355] workqueue mm_percpu_wq: flags=0x8 [ 4345.470558] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4345.478135] pending: vmstat_update [ 4345.484825] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4345.491198] pending: vmstat_update [ 4345.493280] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4345.500184] pending: vmstat_update [ 4345.503358] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4345.506846] pending: vmstat_update [ 4345.508591] workqueue writeback: flags=0x4a [ 4345.510693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4345.514170] pending: wb_workfn, wb_workfn [ 4345.515717] workqueue xprtiod: flags=0xa [ 4345.519021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4345.522455] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4345.529700] workqueue ipv6_addrconf: flags=0x40008 [ 4345.534658] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4345.538151] pending: addrconf_verify_work [ 4345.540354] workqueue knbd0-recv: flags=0x1a [ 4345.542851] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4345.546579] in-flight: 647:recv_work [ 4345.549690] workqueue ptlrpc_pinger: flags=0x2 [ 4345.552140] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4345.558474] pending: ptlrpc_pinger_main [ptlrpc] [ 4345.563081] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4335s workers=2 idle: 652 [ 4352.059350] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4352.060069] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4352.068364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4352.068452] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4352.068460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4352.068463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4352.068476] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4352.068480] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4352.068484] RAX: 0000000000002987 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4352.068487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4352.068489] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4352.068491] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4352.068493] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4352.068496] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4352.068499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4352.068502] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4352.068511] Call Trace: [ 4352.068517] [ 4352.068523] ? show_regs.cold.9+0x22/0x2f [ 4352.068532] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4352.068539] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4352.068543] ? watchdog+0x40/0x40 [ 4352.068547] ? hrtimer_interrupt+0x128/0x2c0 [ 4352.068551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4352.068560] ? apic_timer_interrupt+0xf/0x20 [ 4352.068565] [ 4352.068568] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4352.068574] do_raw_spin_lock+0x177/0x190 [ 4352.072276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4352.072422] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4352.072429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4352.072432] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4352.072446] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4352.072450] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4352.072454] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4352.072455] RDX: 0000000000007d53 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4352.072457] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4352.072459] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4352.072461] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4352.072464] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4352.072466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4352.072594] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4352.072602] Call Trace: [ 4352.072608] [ 4352.072613] ? show_regs.cold.9+0x22/0x2f [ 4352.072622] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4352.072628] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4352.072632] ? watchdog+0x40/0x40 [ 4352.072636] ? hrtimer_interrupt+0x128/0x2c0 [ 4352.072640] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4352.072648] ? apic_timer_interrupt+0xf/0x20 [ 4352.072653] [ 4352.072655] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4352.072660] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4352.072664] do_raw_spin_lock+0x177/0x190 [ 4352.072668] _raw_spin_lock+0x21/0x30 [ 4352.072673] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4352.074687] obd_mod_free+0x46/0x140 [ptlrpc] [ 4352.075194] mdc_commit_open+0x91/0xb0 [mdc] [ 4352.075242] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4352.076021] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4352.077486] after_reply+0xe2c/0x1780 [ptlrpc] [ 4352.125632] _raw_spin_lock+0x21/0x30 [ 4352.125654] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4352.126691] ? do_raw_spin_unlock+0x75/0x190 [ 4352.126700] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4352.126944] ? do_wait_intr+0xf0/0xf0 [ 4352.126953] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4352.127174] mdc_close+0x53b/0xd70 [mdc] [ 4352.127214] lmv_close+0x18d/0x490 [lmv] [ 4352.127246] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4352.127508] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4352.127589] ll_md_close+0x274/0x800 [lustre] [ 4352.146394] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4352.146790] ? __next_timer_interrupt+0x160/0x160 [ 4352.146800] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4352.146999] ? do_wait_intr+0xf0/0xf0 [ 4352.147009] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4352.147217] kthread+0x1d1/0x200 [ 4352.147223] ? set_kthread_struct+0x70/0x70 [ 4352.147227] ret_from_fork+0x1f/0x30 [ 4352.527276] ll_file_release+0x128/0xc50 [lustre] [ 4352.531438] __fput+0x10d/0x440 [ 4352.533479] ____fput+0x12/0x20 [ 4352.542752] task_work_run+0xc6/0x110 [ 4352.545465] exit_to_usermode_loop+0x1dd/0x1f0 [ 4352.551474] do_syscall_64+0x3d6/0x440 [ 4352.559371] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4352.561875] RIP: 0033:0x7f1a63983cd8 [ 4352.573691] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4352.598303] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4352.604344] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4352.607781] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4352.611248] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4352.614430] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4352.620177] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4368.057024] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 4368.062108] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4368.089483] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4368.096952] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4368.101806] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4368.104412] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4368.119960] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4368.123880] RAX: 0000000000006203 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4368.129891] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4368.133141] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4368.138376] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4368.146938] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4368.151799] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4368.158529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4368.163913] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4368.169186] Call Trace: [ 4368.172212] [ 4368.173318] ? show_regs.cold.9+0x22/0x2f [ 4368.174891] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4368.177499] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4368.182251] ? watchdog+0x40/0x40 [ 4368.184635] ? hrtimer_interrupt+0x128/0x2c0 [ 4368.188324] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4368.191296] ? apic_timer_interrupt+0xf/0x20 [ 4368.195885] [ 4368.200028] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4368.210309] do_raw_spin_lock+0x177/0x190 [ 4368.213330] _raw_spin_lock+0x21/0x30 [ 4368.216911] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4368.219149] ? do_raw_spin_unlock+0x75/0x190 [ 4368.224184] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4368.227027] ? do_wait_intr+0xf0/0xf0 [ 4368.229114] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4368.233905] mdc_close+0x53b/0xd70 [mdc] [ 4368.236176] lmv_close+0x18d/0x490 [lmv] [ 4368.238712] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4368.242619] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4368.250093] ll_md_close+0x274/0x800 [lustre] [ 4368.252176] ll_file_release+0x128/0xc50 [lustre] [ 4368.257681] __fput+0x10d/0x440 [ 4368.260086] ____fput+0x12/0x20 [ 4368.262174] task_work_run+0xc6/0x110 [ 4368.266802] exit_to_usermode_loop+0x1dd/0x1f0 [ 4368.271394] do_syscall_64+0x3d6/0x440 [ 4368.274157] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4368.279939] RIP: 0033:0x7f1a63983cd8 [ 4368.281310] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4368.293982] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4368.299199] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4368.303148] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4368.309758] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4368.314187] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4368.318715] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4372.054197] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4372.057883] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4372.079692] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4372.086269] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4372.090838] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4372.093490] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4372.102676] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4372.105823] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4372.109284] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4372.113112] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4372.116439] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4372.119910] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4372.123303] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4372.127431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4372.130168] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4372.133451] Call Trace: [ 4372.134611] [ 4372.136189] ? show_regs.cold.9+0x22/0x2f [ 4372.137814] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4372.140291] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4372.142210] ? watchdog+0x40/0x40 [ 4372.144149] ? hrtimer_interrupt+0x128/0x2c0 [ 4372.145852] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4372.147859] ? apic_timer_interrupt+0xf/0x20 [ 4372.149992] [ 4372.150805] ? native_safe_halt+0x2e/0x50 [ 4372.152410] kvm_wait+0xcd/0xe0 [ 4372.154191] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4372.156628] do_raw_spin_lock+0x177/0x190 [ 4372.158610] _raw_spin_lock+0x21/0x30 [ 4372.160582] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4372.163069] ? do_raw_spin_unlock+0x75/0x190 [ 4372.164885] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4372.167914] ? do_wait_intr+0xf0/0xf0 [ 4372.169809] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4372.172124] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4372.174436] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4372.177705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4372.180212] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4372.183437] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4372.185544] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4372.188126] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4372.190123] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4372.193300] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4372.196062] lmv_intent_open+0x338/0x1150 [lmv] [ 4372.197951] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4372.200934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4372.203098] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4372.206612] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4372.208859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4372.211391] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4372.214459] ? do_raw_spin_unlock+0x75/0x190 [ 4372.216696] ? _raw_spin_unlock+0x12/0x30 [ 4372.218444] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4372.220862] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4372.223152] ll_atomic_open+0x36a/0xd70 [lustre] [ 4372.225615] lookup_open+0x356/0xb40 [ 4372.227355] do_last+0x313/0xfc0 [ 4372.228572] ? nd_jump_root+0xe5/0x160 [ 4372.230061] ? path_init+0x437/0x520 [ 4372.231793] path_openat+0xf7/0x500 [ 4372.233087] ? do_raw_spin_unlock+0x75/0x190 [ 4372.235008] do_filp_open+0x99/0x140 [ 4372.236491] ? getname_flags+0x6e/0x330 [ 4372.237986] ? __check_object_size+0xff/0x256 [ 4372.239991] ? do_raw_spin_unlock+0x75/0x190 [ 4372.242278] ? _raw_spin_unlock+0x12/0x30 [ 4372.244050] do_sys_openat2+0x2b4/0x410 [ 4372.246063] do_sys_open+0x73/0xa0 [ 4372.247364] __x64_sys_openat+0x24/0x30 [ 4372.249088] do_syscall_64+0xc1/0x440 [ 4372.250547] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4372.252452] RIP: 0033:0x7f1a63983332 [ 4372.254815] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4372.263210] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4372.267044] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4372.271366] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4372.274740] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4372.278629] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4372.282200] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4376.033659] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2353s! [ 4376.037527] Showing busy workqueues and worker pools: [ 4376.040958] workqueue events: flags=0x0 [ 4376.044360] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4376.048305] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4376.055878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4376.064898] pending: sec_gc_main [ptlrpc] [ 4376.067263] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4376.073646] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4376.095431] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4376.101365] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4376.109015] workqueue events_long: flags=0x0 [ 4376.112965] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4376.116340] pending: lu_objects_flush [obdclass] [ 4376.119732] workqueue events_unbound: flags=0x2 [ 4376.124758] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4376.127122] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4376.133270] workqueue events_power_efficient: flags=0x80 [ 4376.143017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4376.150702] pending: neigh_periodic_work, do_cache_clean [ 4376.163021] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4376.171719] pending: neigh_periodic_work, check_lifetime [ 4376.178087] workqueue mm_percpu_wq: flags=0x8 [ 4376.183629] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4376.194685] pending: vmstat_update [ 4376.196014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4376.206211] pending: vmstat_update [ 4376.213963] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4376.222254] pending: vmstat_update [ 4376.225108] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4376.233160] pending: vmstat_update [ 4376.239407] workqueue writeback: flags=0x4a [ 4376.246428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4376.262928] pending: wb_workfn, wb_workfn [ 4376.264441] workqueue xprtiod: flags=0xa [ 4376.273896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4376.276302] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4376.293080] workqueue ipv6_addrconf: flags=0x40008 [ 4376.303765] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4376.309927] pending: addrconf_verify_work [ 4376.315143] workqueue knbd0-recv: flags=0x1a [ 4376.318885] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4376.327427] in-flight: 647:recv_work [ 4376.330683] workqueue ptlrpc_pinger: flags=0x2 [ 4376.342822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4376.344931] pending: ptlrpc_pinger_main [ptlrpc] [ 4376.352038] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4366s workers=2 idle: 652 [ 4380.056620] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4380.059024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4380.071686] Modules linked in: lustre(O) [ 4380.078126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4380.078391] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4380.078399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4380.078401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4380.078559] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4380.078564] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4380.078569] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4380.078572] RDX: 0000000000007ddb RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4380.078574] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4380.078576] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4380.078579] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4380.078582] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4380.078585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4380.078587] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4380.078595] Call Trace: [ 4380.078612] [ 4380.078619] ? show_regs.cold.9+0x22/0x2f [ 4380.078629] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4380.078635] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4380.078640] ? watchdog+0x40/0x40 [ 4380.078644] ? hrtimer_interrupt+0x128/0x2c0 [ 4380.078648] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4380.078658] ? apic_timer_interrupt+0xf/0x20 [ 4380.078663] [ 4380.078666] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4380.078671] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4380.078675] do_raw_spin_lock+0x177/0x190 [ 4380.078679] _raw_spin_lock+0x21/0x30 [ 4380.078685] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4380.079065] obd_mod_free+0x46/0x140 [ptlrpc] [ 4380.079569] mdc_commit_open+0x91/0xb0 [mdc] [ 4380.079609] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4380.079807] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4380.080088] after_reply+0xe2c/0x1780 [ptlrpc] [ 4380.080677] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4380.080874] ? __next_timer_interrupt+0x160/0x160 [ 4380.080884] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4380.088426] osp(O) [ 4380.143628] ? do_wait_intr+0xf0/0xf0 [ 4380.143650] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4380.143925] kthread+0x1d1/0x200 [ 4380.143932] ? set_kthread_struct+0x70/0x70 [ 4380.143936] ret_from_fork+0x1f/0x30 [ 4380.345746] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4380.379276] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4380.388645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4380.393173] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4380.396698] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4380.409842] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4380.415899] RAX: 000000000000765c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4380.420506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4380.425604] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4380.430700] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4380.437103] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4380.441245] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4380.446018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4380.451187] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4380.455862] Call Trace: [ 4380.457601] [ 4380.459061] ? show_regs.cold.9+0x22/0x2f [ 4380.461177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4380.463613] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4380.465606] ? watchdog+0x40/0x40 [ 4380.467336] ? hrtimer_interrupt+0x128/0x2c0 [ 4380.470029] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4380.472071] ? apic_timer_interrupt+0xf/0x20 [ 4380.474791] [ 4380.475663] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4380.477917] do_raw_spin_lock+0x177/0x190 [ 4380.479603] _raw_spin_lock+0x21/0x30 [ 4380.481372] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4380.483938] ? do_raw_spin_unlock+0x75/0x190 [ 4380.485805] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4380.488097] ? do_wait_intr+0xf0/0xf0 [ 4380.489423] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4380.491632] mdc_close+0x53b/0xd70 [mdc] [ 4380.493560] lmv_close+0x18d/0x490 [lmv] [ 4380.495386] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4380.498232] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4380.501138] ll_md_close+0x274/0x800 [lustre] [ 4380.503338] ll_file_release+0x128/0xc50 [lustre] [ 4380.506231] __fput+0x10d/0x440 [ 4380.508339] ____fput+0x12/0x20 [ 4380.510581] task_work_run+0xc6/0x110 [ 4380.512852] exit_to_usermode_loop+0x1dd/0x1f0 [ 4380.515094] do_syscall_64+0x3d6/0x440 [ 4380.518393] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4380.521828] RIP: 0033:0x7f1a63983cd8 [ 4380.523497] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4380.532294] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4380.538587] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4380.542082] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4380.546412] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4380.550297] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4380.554928] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4396.057754] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 4396.064219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4396.106611] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4396.113313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4396.127826] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4396.130717] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4396.147646] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4396.151644] RAX: 0000000000007b84 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4396.155391] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4396.160041] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4396.166829] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4396.173937] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4396.183192] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4396.191166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4396.198747] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4396.205213] Call Trace: [ 4396.206874] [ 4396.208301] ? show_regs.cold.9+0x22/0x2f [ 4396.212738] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4396.215818] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4396.218358] ? watchdog+0x40/0x40 [ 4396.220259] ? hrtimer_interrupt+0x128/0x2c0 [ 4396.222712] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4396.225200] ? apic_timer_interrupt+0xf/0x20 [ 4396.227211] [ 4396.229181] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4396.232232] do_raw_spin_lock+0x177/0x190 [ 4396.234849] _raw_spin_lock+0x21/0x30 [ 4396.237057] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4396.240763] ? do_raw_spin_unlock+0x75/0x190 [ 4396.244050] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4396.246398] ? do_wait_intr+0xf0/0xf0 [ 4396.247812] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4396.251096] mdc_close+0x53b/0xd70 [mdc] [ 4396.256732] lmv_close+0x18d/0x490 [lmv] [ 4396.260255] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4396.264105] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4396.270540] ll_md_close+0x274/0x800 [lustre] [ 4396.272814] ll_file_release+0x128/0xc50 [lustre] [ 4396.278221] __fput+0x10d/0x440 [ 4396.280062] ____fput+0x12/0x20 [ 4396.282207] task_work_run+0xc6/0x110 [ 4396.283457] exit_to_usermode_loop+0x1dd/0x1f0 [ 4396.284980] do_syscall_64+0x3d6/0x440 [ 4396.287034] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4396.289924] RIP: 0033:0x7f1a63983cd8 [ 4396.292464] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4396.305899] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4396.313664] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4396.320346] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4396.326021] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4396.330519] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4396.334086] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4404.053117] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4404.057806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4404.086492] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4404.094022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4404.101699] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4404.106118] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4404.117932] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4404.122349] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4404.126671] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4404.130694] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4404.134958] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4404.139675] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4404.143000] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4404.147141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4404.150728] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4404.154723] Call Trace: [ 4404.156642] [ 4404.157775] ? show_regs.cold.9+0x22/0x2f [ 4404.160000] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4404.162695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4404.165828] ? watchdog+0x40/0x40 [ 4404.168434] ? hrtimer_interrupt+0x128/0x2c0 [ 4404.170690] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4404.173520] ? apic_timer_interrupt+0xf/0x20 [ 4404.175685] [ 4404.177086] ? native_safe_halt+0x2e/0x50 [ 4404.179057] kvm_wait+0xcd/0xe0 [ 4404.181021] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4404.183543] do_raw_spin_lock+0x177/0x190 [ 4404.185438] _raw_spin_lock+0x21/0x30 [ 4404.187396] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4404.190036] ? do_raw_spin_unlock+0x75/0x190 [ 4404.192089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4404.195210] ? do_wait_intr+0xf0/0xf0 [ 4404.197109] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4404.199998] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4404.202989] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4404.206643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4404.209990] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4404.212750] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4404.215176] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4404.217598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4404.219995] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4404.223599] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4404.226959] lmv_intent_open+0x338/0x1150 [lmv] [ 4404.229442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4404.232474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4404.235012] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4404.237642] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4404.240117] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4404.243021] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4404.245111] ? do_raw_spin_unlock+0x75/0x190 [ 4404.246861] ? _raw_spin_unlock+0x12/0x30 [ 4404.249991] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4404.253172] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4404.256729] ll_atomic_open+0x36a/0xd70 [lustre] [ 4404.259715] lookup_open+0x356/0xb40 [ 4404.261242] do_last+0x313/0xfc0 [ 4404.263028] ? nd_jump_root+0xe5/0x160 [ 4404.264992] ? path_init+0x437/0x520 [ 4404.266980] path_openat+0xf7/0x500 [ 4404.269229] ? do_raw_spin_unlock+0x75/0x190 [ 4404.271295] do_filp_open+0x99/0x140 [ 4404.274317] ? getname_flags+0x6e/0x330 [ 4404.276187] ? __check_object_size+0xff/0x256 [ 4404.279271] ? do_raw_spin_unlock+0x75/0x190 [ 4404.281979] ? _raw_spin_unlock+0x12/0x30 [ 4404.283704] do_sys_openat2+0x2b4/0x410 [ 4404.287869] do_sys_open+0x73/0xa0 [ 4404.290680] __x64_sys_openat+0x24/0x30 [ 4404.294333] do_syscall_64+0xc1/0x440 [ 4404.297392] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4404.300596] RIP: 0033:0x7f1a63983332 [ 4404.302811] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4404.312173] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4404.316333] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4404.320117] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4404.324016] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4404.327878] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4404.331559] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4406.757933] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2383s! [ 4406.772575] Showing busy workqueues and worker pools: [ 4406.778865] workqueue events: flags=0x0 [ 4406.782883] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4406.790954] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4406.802636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4406.807800] pending: sec_gc_main [ptlrpc] [ 4406.810940] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4406.814871] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4406.823637] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4406.827101] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4406.834141] workqueue events_long: flags=0x0 [ 4406.837506] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4406.845827] pending: lu_objects_flush [obdclass] [ 4406.852670] workqueue events_unbound: flags=0x2 [ 4406.861702] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4406.866915] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4406.873986] workqueue events_power_efficient: flags=0x80 [ 4406.882321] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4406.889080] pending: neigh_periodic_work, do_cache_clean [ 4406.893753] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4406.903075] pending: neigh_periodic_work, check_lifetime [ 4406.909655] workqueue mm_percpu_wq: flags=0x8 [ 4406.913824] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4406.922667] pending: vmstat_update [ 4406.930878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4406.935736] pending: vmstat_update [ 4406.938697] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4406.942862] pending: vmstat_update [ 4406.946869] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4406.955955] pending: vmstat_update [ 4406.967812] workqueue writeback: flags=0x4a [ 4406.972047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4406.978838] pending: wb_workfn, wb_workfn [ 4406.992897] workqueue xprtiod: flags=0xa [ 4406.994499] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4407.002547] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4407.013367] workqueue ipv6_addrconf: flags=0x40008 [ 4407.019816] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4407.025888] pending: addrconf_verify_work [ 4407.034939] workqueue knbd0-recv: flags=0x1a [ 4407.036649] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4407.050014] in-flight: 647:recv_work [ 4407.055836] workqueue ptlrpc_pinger: flags=0x2 [ 4407.059127] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4407.064816] pending: ptlrpc_pinger_main [ptlrpc] [ 4407.073916] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4397s workers=2 idle: 652 [ 4408.057342] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4408.058058] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4408.064474] Modules linked in: [ 4408.071115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4408.071752] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4408.071759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4408.071763] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4408.071777] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4408.071781] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4408.071785] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4408.071788] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4408.071789] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4408.071791] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4408.071793] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4408.071796] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4408.071798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4408.071801] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4408.071810] Call Trace: [ 4408.071815] [ 4408.071831] ? show_regs.cold.9+0x22/0x2f [ 4408.071842] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4408.071849] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4408.071853] ? watchdog+0x40/0x40 [ 4408.071857] ? hrtimer_interrupt+0x128/0x2c0 [ 4408.071861] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4408.071867] ? apic_timer_interrupt+0xf/0x20 [ 4408.071872] [ 4408.071875] ? native_safe_halt+0x2e/0x50 [ 4408.071879] kvm_wait+0xcd/0xe0 [ 4408.071886] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4408.071894] do_raw_spin_lock+0x177/0x190 [ 4408.071899] _raw_spin_lock+0x21/0x30 [ 4408.071903] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4408.072379] obd_mod_free+0x46/0x140 [ptlrpc] [ 4408.073789] lustre(O) [ 4408.122135] mdc_commit_open+0x91/0xb0 [mdc] [ 4408.122324] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4408.122583] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4408.123137] after_reply+0xe2c/0x1780 [ptlrpc] [ 4408.123607] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4408.142859] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4408.142914] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4408.142920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4408.142923] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4408.149078] ? __next_timer_interrupt+0x160/0x160 [ 4408.150283] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4408.150292] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4408.150299] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4408.150301] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4408.150303] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4408.150304] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4408.150306] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4408.150309] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4408.150312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4408.150314] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4408.150321] Call Trace: [ 4408.150324] [ 4408.150328] ? show_regs.cold.9+0x22/0x2f [ 4408.150337] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4408.150343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4408.150347] ? watchdog+0x40/0x40 [ 4408.150352] ? hrtimer_interrupt+0x128/0x2c0 [ 4408.150355] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4408.150363] ? apic_timer_interrupt+0xf/0x20 [ 4408.150367] [ 4408.150370] ? native_safe_halt+0x2e/0x50 [ 4408.150374] kvm_wait+0xcd/0xe0 [ 4408.150383] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4408.150392] do_raw_spin_lock+0x177/0x190 [ 4408.150397] _raw_spin_lock+0x21/0x30 [ 4408.150401] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4408.150663] ? do_raw_spin_unlock+0x75/0x190 [ 4408.150669] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4408.151250] ? do_wait_intr+0xf0/0xf0 [ 4408.151258] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4408.151457] mdc_close+0x53b/0xd70 [mdc] [ 4408.151493] lmv_close+0x18d/0x490 [lmv] [ 4408.151516] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4408.151623] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4408.151707] ll_md_close+0x274/0x800 [lustre] [ 4408.151791] ll_file_release+0x128/0xc50 [lustre] [ 4408.152002] __fput+0x10d/0x440 [ 4408.152011] ____fput+0x12/0x20 [ 4408.152015] task_work_run+0xc6/0x110 [ 4408.152023] exit_to_usermode_loop+0x1dd/0x1f0 [ 4408.152031] do_syscall_64+0x3d6/0x440 [ 4408.152036] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4408.152044] RIP: 0033:0x7f1a63983cd8 [ 4408.152050] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4408.152054] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4408.152059] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4408.152061] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4408.152063] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4408.152065] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4408.152067] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4408.769558] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4408.771097] ? do_wait_intr+0xf0/0xf0 [ 4408.772493] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4408.774544] kthread+0x1d1/0x200 [ 4408.775566] ? set_kthread_struct+0x70/0x70 [ 4408.777394] ret_from_fork+0x1f/0x30 [ 4422.464028] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4422.469818] rcu: 3-...!: (1442310 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 4422.473370] (detected by 1, t=2400058 jiffies, g=269633, q=977) [ 4422.476177] Sending NMI from CPU 1 to CPUs 3: [ 4422.478966] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 4422.479434] rcu: rcu_sched kthread starved for 2399704 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 4422.491319] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4422.496587] rcu: RCU grace-period kthread stack dump: [ 4422.499178] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4422.506991] Call Trace: [ 4422.507947] __schedule+0x351/0xcb0 [ 4422.509550] schedule+0xc0/0x180 [ 4422.511959] schedule_timeout+0xb4/0x190 [ 4422.513921] ? do_raw_spin_unlock+0x75/0x190 [ 4422.516605] ? __next_timer_interrupt+0x160/0x160 [ 4422.519623] ? prepare_to_swait_event+0xf3/0x140 [ 4422.521733] rcu_gp_kthread+0x8dd/0x15b0 [ 4422.523294] ? force_qs_rnp+0x2c0/0x2c0 [ 4422.524833] kthread+0x1d1/0x200 [ 4422.526523] ? set_kthread_struct+0x70/0x70 [ 4422.530162] ret_from_fork+0x1f/0x30 [ 4422.531966] rcu: Stack dump where RCU GP kthread last ran: [ 4422.534012] NMI backtrace for cpu 1 [ 4422.535275] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4422.540983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4422.563470] Call Trace: [ 4422.564316] [ 4422.565001] ? dump_stack+0xbb/0x10e [ 4422.569656] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4422.571465] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4422.573734] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4422.577117] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4422.580505] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4422.583158] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 4422.589797] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 4422.597551] ? update_process_times+0x6d/0xc0 [ 4422.602336] ? tick_sched_handle.isra.18+0x27/0x90 [ 4422.604003] ? tick_sched_timer+0x53/0xb0 [ 4422.608372] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4422.611301] ? tick_sched_handle.isra.18+0x90/0x90 [ 4422.612684] ? hrtimer_interrupt+0x128/0x2c0 [ 4422.616509] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4422.618449] ? apic_timer_interrupt+0xf/0x20 [ 4422.620455] [ 4422.621167] ? native_safe_halt+0x2e/0x50 [ 4422.623208] ? kvm_wait+0xcd/0xe0 [ 4422.624564] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4422.627004] ? do_raw_spin_lock+0x177/0x190 [ 4422.628936] ? _raw_spin_lock+0x21/0x30 [ 4422.630848] ? ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4422.633324] ? do_raw_spin_unlock+0x75/0x190 [ 4422.634892] ? ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4422.637144] ? do_wait_intr+0xf0/0xf0 [ 4422.639237] ? ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4422.641429] ? mdc_close+0x53b/0xd70 [mdc] [ 4422.642882] ? lmv_close+0x18d/0x490 [lmv] [ 4422.644693] ? ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4422.647213] ? ll_md_real_close+0xa5/0x2a0 [lustre] [ 4422.649233] ? ll_md_close+0x274/0x800 [lustre] [ 4422.651182] ? ll_file_release+0x128/0xc50 [lustre] [ 4422.653276] ? __fput+0x10d/0x440 [ 4422.654612] ? ____fput+0x12/0x20 [ 4422.655957] ? task_work_run+0xc6/0x110 [ 4422.657397] ? exit_to_usermode_loop+0x1dd/0x1f0 [ 4422.659342] ? do_syscall_64+0x3d6/0x440 [ 4422.660730] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4424.057975] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 4424.069958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4424.137964] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4424.149875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4424.153964] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4424.159242] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4424.183265] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4424.195081] RAX: 0000000000007f84 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4424.197858] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4424.208561] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4424.213806] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4424.221796] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4424.226117] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4424.233207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4424.239755] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4424.249830] Call Trace: [ 4424.252960] [ 4424.257369] ? show_regs.cold.9+0x22/0x2f [ 4424.261249] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4424.266540] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4424.279224] ? watchdog+0x40/0x40 [ 4424.285826] ? hrtimer_interrupt+0x128/0x2c0 [ 4424.292862] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4424.309474] ? apic_timer_interrupt+0xf/0x20 [ 4424.311201] [ 4424.311951] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4424.332032] do_raw_spin_lock+0x177/0x190 [ 4424.335486] _raw_spin_lock+0x21/0x30 [ 4424.338453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4424.347863] ? do_raw_spin_unlock+0x75/0x190 [ 4424.350514] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4424.354774] ? do_wait_intr+0xf0/0xf0 [ 4424.358817] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4424.365621] mdc_close+0x53b/0xd70 [mdc] [ 4424.371209] lmv_close+0x18d/0x490 [lmv] [ 4424.373604] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4424.387097] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4424.392721] ll_md_close+0x274/0x800 [lustre] [ 4424.398459] ll_file_release+0x128/0xc50 [lustre] [ 4424.409814] __fput+0x10d/0x440 [ 4424.414573] ____fput+0x12/0x20 [ 4424.418352] task_work_run+0xc6/0x110 [ 4424.421709] exit_to_usermode_loop+0x1dd/0x1f0 [ 4424.425589] do_syscall_64+0x3d6/0x440 [ 4424.426709] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4424.431350] RIP: 0033:0x7f1a63983cd8 [ 4424.435909] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4424.450310] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4424.460657] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4424.468253] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4424.471137] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4424.487971] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4424.497390] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4432.053149] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4432.063471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4432.120269] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4432.133150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4432.147478] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4432.152570] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4432.173890] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4432.181032] RAX: 0000000000004a90 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4432.185259] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4432.189415] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4432.192833] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4432.196111] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4432.201091] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4432.204897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4432.208499] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4432.212583] Call Trace: [ 4432.214454] [ 4432.215412] ? show_regs.cold.9+0x22/0x2f [ 4432.218420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4432.225834] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4432.230561] ? watchdog+0x40/0x40 [ 4432.237727] ? hrtimer_interrupt+0x128/0x2c0 [ 4432.243844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4432.247242] ? apic_timer_interrupt+0xf/0x20 [ 4432.251719] [ 4432.252840] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4432.258638] do_raw_spin_lock+0x177/0x190 [ 4432.261525] _raw_spin_lock+0x21/0x30 [ 4432.264214] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4432.278330] ? do_raw_spin_unlock+0x75/0x190 [ 4432.283326] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4432.294246] ? do_wait_intr+0xf0/0xf0 [ 4432.299773] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4432.309774] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4432.311765] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4432.318662] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4432.325672] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4432.330336] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4432.334870] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4432.339634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4432.342574] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4432.349208] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4432.360464] lmv_intent_open+0x338/0x1150 [lmv] [ 4432.366593] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4432.375174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4432.381121] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4432.387748] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4432.396503] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4432.406334] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4432.414682] ? do_raw_spin_unlock+0x75/0x190 [ 4432.420329] ? _raw_spin_unlock+0x12/0x30 [ 4432.429336] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4432.435818] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4432.445159] ll_atomic_open+0x36a/0xd70 [lustre] [ 4432.452308] lookup_open+0x356/0xb40 [ 4432.457144] do_last+0x313/0xfc0 [ 4432.459236] ? nd_jump_root+0xe5/0x160 [ 4432.465905] ? path_init+0x437/0x520 [ 4432.471872] path_openat+0xf7/0x500 [ 4432.475068] ? do_raw_spin_unlock+0x75/0x190 [ 4432.481568] do_filp_open+0x99/0x140 [ 4432.485254] ? getname_flags+0x6e/0x330 [ 4432.490490] ? __check_object_size+0xff/0x256 [ 4432.495070] ? do_raw_spin_unlock+0x75/0x190 [ 4432.515082] ? _raw_spin_unlock+0x12/0x30 [ 4432.524142] do_sys_openat2+0x2b4/0x410 [ 4432.530911] do_sys_open+0x73/0xa0 [ 4432.540580] __x64_sys_openat+0x24/0x30 [ 4432.551400] do_syscall_64+0xc1/0x440 [ 4432.558356] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4432.565230] RIP: 0033:0x7f1a63983332 [ 4432.567257] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4432.575973] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4432.579911] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4432.582903] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4432.586051] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4432.590107] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4432.594325] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4437.478034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2414s! [ 4437.481514] Showing busy workqueues and worker pools: [ 4437.483585] workqueue events: flags=0x0 [ 4437.485188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4437.488002] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4437.491091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4437.495710] pending: sec_gc_main [ptlrpc] [ 4437.498035] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4437.500843] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4437.505521] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4437.508267] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4437.512243] workqueue events_long: flags=0x0 [ 4437.513761] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4437.516231] pending: lu_objects_flush [obdclass] [ 4437.519201] workqueue events_unbound: flags=0x2 [ 4437.520878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4437.523552] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4437.528012] workqueue events_power_efficient: flags=0x80 [ 4437.530015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4437.532757] pending: neigh_periodic_work, do_cache_clean [ 4437.534852] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4437.537872] pending: neigh_periodic_work, check_lifetime [ 4437.540350] workqueue mm_percpu_wq: flags=0x8 [ 4437.542075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4437.544827] pending: vmstat_update [ 4437.546548] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4437.549172] pending: vmstat_update [ 4437.550649] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4437.553001] pending: vmstat_update [ 4437.554520] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4437.557583] pending: vmstat_update [ 4437.559116] workqueue writeback: flags=0x4a [ 4437.560795] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4437.563245] pending: wb_workfn, wb_workfn [ 4437.565648] workqueue xprtiod: flags=0xa [ 4437.567727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4437.570181] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4437.574017] workqueue ipv6_addrconf: flags=0x40008 [ 4437.575993] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4437.578641] pending: addrconf_verify_work [ 4437.580756] workqueue knbd0-recv: flags=0x1a [ 4437.582528] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4437.585158] in-flight: 647:recv_work [ 4437.586795] workqueue ptlrpc_pinger: flags=0x2 [ 4437.588868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4437.591554] pending: ptlrpc_pinger_main [ptlrpc] [ 4437.594059] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4427s workers=2 idle: 652 [ 4448.056596] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4448.058109] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4448.066881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 4448.070764] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4448.070863] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4448.070871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4448.070874] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4448.070888] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4448.070892] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4448.070896] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4448.070898] RDX: 0000000000006e5c RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4448.070901] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4448.070903] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4448.070905] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4448.070907] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4448.070910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4448.070912] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4448.070921] Call Trace: [ 4448.070926] [ 4448.070933] ? show_regs.cold.9+0x22/0x2f [ 4448.070942] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4448.070949] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4448.070953] ? watchdog+0x40/0x40 [ 4448.070956] ? hrtimer_interrupt+0x128/0x2c0 [ 4448.070960] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4448.070970] ? apic_timer_interrupt+0xf/0x20 [ 4448.070975] [ 4448.070978] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4448.070983] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4448.070987] do_raw_spin_lock+0x177/0x190 [ 4448.070992] _raw_spin_lock+0x21/0x30 [ 4448.070997] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4448.071515] obd_mod_free+0x46/0x140 [ptlrpc] [ 4448.071871] mdc_commit_open+0x91/0xb0 [mdc] [ 4448.071915] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4448.072194] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4448.072452] after_reply+0xe2c/0x1780 [ptlrpc] [ 4448.072796] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4448.073021] ? __next_timer_interrupt+0x160/0x160 [ 4448.073097] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4448.073324] ? do_wait_intr+0xf0/0xf0 [ 4448.073333] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4448.093570] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic [ 4448.111436] kthread+0x1d1/0x200 [ 4448.111456] ? set_kthread_struct+0x70/0x70 [ 4448.111461] ret_from_fork+0x1f/0x30 [ 4448.282740] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4448.293321] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4448.299018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4448.309198] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4448.318691] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4448.339894] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4448.349199] RAX: 000000000000166a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4448.355902] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4448.361312] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4448.365516] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4448.367676] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4448.378950] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4448.386700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4448.390658] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4448.394576] Call Trace: [ 4448.396069] [ 4448.397064] ? show_regs.cold.9+0x22/0x2f [ 4448.399298] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4448.409547] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4448.412055] ? watchdog+0x40/0x40 [ 4448.415978] ? hrtimer_interrupt+0x128/0x2c0 [ 4448.418759] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4448.426591] ? apic_timer_interrupt+0xf/0x20 [ 4448.429046] [ 4448.429682] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4448.432031] do_raw_spin_lock+0x177/0x190 [ 4448.434282] _raw_spin_lock+0x21/0x30 [ 4448.439705] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4448.442605] ? do_raw_spin_unlock+0x75/0x190 [ 4448.443902] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4448.453467] ? do_wait_intr+0xf0/0xf0 [ 4448.454436] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4448.462296] mdc_close+0x53b/0xd70 [mdc] [ 4448.465436] lmv_close+0x18d/0x490 [lmv] [ 4448.469304] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4448.478578] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4448.481414] ll_md_close+0x274/0x800 [lustre] [ 4448.487334] ll_file_release+0x128/0xc50 [lustre] [ 4448.494235] __fput+0x10d/0x440 [ 4448.500260] ____fput+0x12/0x20 [ 4448.502632] task_work_run+0xc6/0x110 [ 4448.507760] exit_to_usermode_loop+0x1dd/0x1f0 [ 4448.516297] do_syscall_64+0x3d6/0x440 [ 4448.518417] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4448.520430] RIP: 0033:0x7f1a63983cd8 [ 4448.521744] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4448.541456] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4448.545619] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4448.548567] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4448.562568] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4448.567089] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4448.572030] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4452.058147] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 4452.067406] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4452.124291] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4452.129686] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4452.134894] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4452.136955] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4452.149732] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4452.156686] RAX: 0000000000005a8c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4452.162120] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4452.165536] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4452.169031] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4452.191325] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4452.199436] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4452.206634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4452.211730] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4452.216519] Call Trace: [ 4452.217348] [ 4452.220637] ? show_regs.cold.9+0x22/0x2f [ 4452.226619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4452.228652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4452.235691] ? watchdog+0x40/0x40 [ 4452.237059] ? hrtimer_interrupt+0x128/0x2c0 [ 4452.241640] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4452.247091] ? apic_timer_interrupt+0xf/0x20 [ 4452.249301] [ 4452.250814] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4452.255430] do_raw_spin_lock+0x177/0x190 [ 4452.260474] _raw_spin_lock+0x21/0x30 [ 4452.263713] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4452.269494] ? do_raw_spin_unlock+0x75/0x190 [ 4452.273487] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4452.277152] ? do_wait_intr+0xf0/0xf0 [ 4452.279060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4452.282724] mdc_close+0x53b/0xd70 [mdc] [ 4452.287402] lmv_close+0x18d/0x490 [lmv] [ 4452.291071] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4452.295246] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4452.297827] ll_md_close+0x274/0x800 [lustre] [ 4452.299887] ll_file_release+0x128/0xc50 [lustre] [ 4452.302848] __fput+0x10d/0x440 [ 4452.304218] ____fput+0x12/0x20 [ 4452.305662] task_work_run+0xc6/0x110 [ 4452.307307] exit_to_usermode_loop+0x1dd/0x1f0 [ 4452.309435] do_syscall_64+0x3d6/0x440 [ 4452.311215] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4452.313475] RIP: 0033:0x7f1a63983cd8 [ 4452.315070] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4452.322915] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4452.325413] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4452.327531] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4452.330163] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4452.335478] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4452.341669] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4464.056915] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4464.063293] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4464.094568] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4464.110278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4464.121123] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4464.124569] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4464.137784] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4464.147088] RAX: 0000000000007d92 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4464.154841] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4464.159313] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4464.167945] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4464.173834] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4464.182346] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4464.187027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4464.190584] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4464.195165] Call Trace: [ 4464.196207] [ 4464.197153] ? show_regs.cold.9+0x22/0x2f [ 4464.199514] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4464.202172] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4464.203730] ? watchdog+0x40/0x40 [ 4464.205475] ? hrtimer_interrupt+0x128/0x2c0 [ 4464.211561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4464.214261] ? apic_timer_interrupt+0xf/0x20 [ 4464.220388] [ 4464.221572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4464.226849] do_raw_spin_lock+0x177/0x190 [ 4464.230850] _raw_spin_lock+0x21/0x30 [ 4464.233850] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4464.236615] ? do_raw_spin_unlock+0x75/0x190 [ 4464.241456] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4464.245752] ? do_wait_intr+0xf0/0xf0 [ 4464.247908] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4464.250268] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4464.253322] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4464.255929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4464.258073] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4464.260162] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4464.262039] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4464.267448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4464.269909] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4464.274989] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4464.278057] lmv_intent_open+0x338/0x1150 [lmv] [ 4464.279990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4464.283321] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4464.285748] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4464.288843] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4464.293058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4464.297404] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4464.300021] ? do_raw_spin_unlock+0x75/0x190 [ 4464.302398] ? _raw_spin_unlock+0x12/0x30 [ 4464.304223] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4464.311384] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4464.314330] ll_atomic_open+0x36a/0xd70 [lustre] [ 4464.317238] lookup_open+0x356/0xb40 [ 4464.318998] do_last+0x313/0xfc0 [ 4464.322194] ? nd_jump_root+0xe5/0x160 [ 4464.325158] ? path_init+0x437/0x520 [ 4464.327712] path_openat+0xf7/0x500 [ 4464.329408] ? do_raw_spin_unlock+0x75/0x190 [ 4464.332505] do_filp_open+0x99/0x140 [ 4464.334335] ? getname_flags+0x6e/0x330 [ 4464.336786] ? __check_object_size+0xff/0x256 [ 4464.340975] ? do_raw_spin_unlock+0x75/0x190 [ 4464.344511] ? _raw_spin_unlock+0x12/0x30 [ 4464.345898] do_sys_openat2+0x2b4/0x410 [ 4464.349260] do_sys_open+0x73/0xa0 [ 4464.350912] __x64_sys_openat+0x24/0x30 [ 4464.352599] do_syscall_64+0xc1/0x440 [ 4464.354519] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4464.357244] RIP: 0033:0x7f1a63983332 [ 4464.359058] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4464.379995] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4464.383222] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4464.387234] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4464.391383] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4464.395575] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4464.399921] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4467.680057] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2444s! [ 4467.682639] Showing busy workqueues and worker pools: [ 4467.684152] workqueue events: flags=0x0 [ 4467.685296] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4467.691830] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4467.699731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4467.709936] pending: sec_gc_main [ptlrpc] [ 4467.711451] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4467.731533] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4467.737556] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4467.743349] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4467.753269] workqueue events_long: flags=0x0 [ 4467.758567] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4467.762910] pending: lu_objects_flush [obdclass] [ 4467.769430] workqueue events_unbound: flags=0x2 [ 4467.772649] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4467.780084] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4467.788202] workqueue events_power_efficient: flags=0x80 [ 4467.790495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4467.796983] pending: neigh_periodic_work, do_cache_clean [ 4467.801177] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4467.806132] pending: neigh_periodic_work, check_lifetime [ 4467.809792] workqueue mm_percpu_wq: flags=0x8 [ 4467.812772] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4467.816720] pending: vmstat_update [ 4467.819115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4467.823283] pending: vmstat_update [ 4467.825330] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4467.830692] pending: vmstat_update [ 4467.833587] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4467.837273] pending: vmstat_update [ 4467.838971] workqueue writeback: flags=0x4a [ 4467.847114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4467.853473] pending: wb_workfn, wb_workfn [ 4467.863791] workqueue xprtiod: flags=0xa [ 4467.865364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4467.871130] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4467.877857] workqueue ipv6_addrconf: flags=0x40008 [ 4467.881740] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4467.885508] pending: addrconf_verify_work [ 4467.887972] workqueue knbd0-recv: flags=0x1a [ 4467.889708] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4467.896889] in-flight: 647:recv_work [ 4467.900143] workqueue ptlrpc_pinger: flags=0x2 [ 4467.901584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4467.907108] pending: ptlrpc_pinger_main [ptlrpc] [ 4467.910552] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4457s workers=2 idle: 652 [ 4476.056020] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4476.058490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4476.058594] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4476.078496] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4476.082251] Modules linked in: [ 4476.087128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4476.087133] lustre(O) osp(O) ofd(O) [ 4476.088617] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4476.092618] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4476.092692] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4476.092700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4476.092703] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4476.092715] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4476.092720] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4476.092725] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4476.092727] RDX: 0000000000005520 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4476.092730] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4476.092732] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4476.092734] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4476.092738] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4476.092740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4476.092743] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4476.092752] Call Trace: [ 4476.092757] [ 4476.092763] ? show_regs.cold.9+0x22/0x2f [ 4476.092772] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4476.092779] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4476.092783] ? watchdog+0x40/0x40 [ 4476.092788] ? hrtimer_interrupt+0x128/0x2c0 [ 4476.092792] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4476.092800] ? apic_timer_interrupt+0xf/0x20 [ 4476.092805] [ 4476.092808] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4476.092813] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4476.092818] do_raw_spin_lock+0x177/0x190 [ 4476.092823] _raw_spin_lock+0x21/0x30 [ 4476.092828] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4476.093110] obd_mod_free+0x46/0x140 [ptlrpc] [ 4476.093388] mdc_commit_open+0x91/0xb0 [mdc] [ 4476.093500] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4476.093791] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4476.094007] after_reply+0xe2c/0x1780 [ptlrpc] [ 4476.094201] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4476.096099] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4476.096107] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4476.096113] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4476.096115] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4476.096117] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4476.096119] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4476.096122] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4476.096125] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4476.096128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4476.096130] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4476.096138] Call Trace: [ 4476.096143] [ 4476.096147] ? show_regs.cold.9+0x22/0x2f [ 4476.096159] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4476.096165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4476.096170] ? watchdog+0x40/0x40 [ 4476.096174] ? hrtimer_interrupt+0x128/0x2c0 [ 4476.096178] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4476.096186] ? apic_timer_interrupt+0xf/0x20 [ 4476.096191] [ 4476.096194] ? native_safe_halt+0x2e/0x50 [ 4476.096199] kvm_wait+0xcd/0xe0 [ 4476.096205] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4476.096214] do_raw_spin_lock+0x177/0x190 [ 4476.096218] _raw_spin_lock+0x21/0x30 [ 4476.096222] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4476.096719] ? do_raw_spin_unlock+0x75/0x190 [ 4476.096725] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4476.096919] ? do_wait_intr+0xf0/0xf0 [ 4476.096926] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4476.097128] mdc_close+0x53b/0xd70 [mdc] [ 4476.097167] lmv_close+0x18d/0x490 [lmv] [ 4476.097198] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4476.099339] ? __next_timer_interrupt+0x160/0x160 [ 4476.099354] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4476.099652] ? do_wait_intr+0xf0/0xf0 [ 4476.099661] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4476.099859] kthread+0x1d1/0x200 [ 4476.099864] ? set_kthread_struct+0x70/0x70 [ 4476.099868] ret_from_fork+0x1f/0x30 [ 4476.432763] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4476.437179] ll_md_close+0x274/0x800 [lustre] [ 4476.441147] ll_file_release+0x128/0xc50 [lustre] [ 4476.445987] __fput+0x10d/0x440 [ 4476.448619] ____fput+0x12/0x20 [ 4476.450932] task_work_run+0xc6/0x110 [ 4476.453272] exit_to_usermode_loop+0x1dd/0x1f0 [ 4476.457284] do_syscall_64+0x3d6/0x440 [ 4476.461154] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4476.462964] RIP: 0033:0x7f1a63983cd8 [ 4476.466230] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4476.482939] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4476.489092] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4476.494983] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4476.501027] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4476.506927] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4476.512850] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4480.057792] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4480.067783] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4480.147051] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4480.168486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4480.177904] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4480.187390] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4480.220647] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4480.234785] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4480.247659] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4480.250608] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4480.267109] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4480.275667] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4480.291159] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4480.300013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4480.307205] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4480.315207] Call Trace: [ 4480.317930] [ 4480.318917] ? show_regs.cold.9+0x22/0x2f [ 4480.325169] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4480.331154] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4480.335714] ? watchdog+0x40/0x40 [ 4480.343061] ? hrtimer_interrupt+0x128/0x2c0 [ 4480.344492] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4480.351368] ? apic_timer_interrupt+0xf/0x20 [ 4480.354347] [ 4480.357134] ? native_safe_halt+0x2e/0x50 [ 4480.363867] kvm_wait+0xcd/0xe0 [ 4480.364762] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4480.366084] do_raw_spin_lock+0x177/0x190 [ 4480.372212] _raw_spin_lock+0x21/0x30 [ 4480.375856] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4480.378242] ? do_raw_spin_unlock+0x75/0x190 [ 4480.381916] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4480.391934] ? do_wait_intr+0xf0/0xf0 [ 4480.397676] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4480.402800] mdc_close+0x53b/0xd70 [mdc] [ 4480.404512] lmv_close+0x18d/0x490 [lmv] [ 4480.409604] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4480.422043] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4480.429792] ll_md_close+0x274/0x800 [lustre] [ 4480.436382] ll_file_release+0x128/0xc50 [lustre] [ 4480.444682] __fput+0x10d/0x440 [ 4480.447164] ____fput+0x12/0x20 [ 4480.448193] task_work_run+0xc6/0x110 [ 4480.455203] exit_to_usermode_loop+0x1dd/0x1f0 [ 4480.462386] do_syscall_64+0x3d6/0x440 [ 4480.468019] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4480.475653] RIP: 0033:0x7f1a63983cd8 [ 4480.478646] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4480.497433] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4480.505250] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4480.517898] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4480.526742] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4480.529906] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4480.533450] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4492.054185] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4492.062931] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4492.132088] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4492.150865] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4492.163888] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4492.175939] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4492.199863] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4492.209347] RAX: 000000000000780e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4492.217691] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4492.226440] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4492.234522] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4492.243705] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4492.251897] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4492.260537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4492.266953] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4492.275881] Call Trace: [ 4492.276728] [ 4492.278901] ? show_regs.cold.9+0x22/0x2f [ 4492.285243] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4492.291885] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4492.294230] ? watchdog+0x40/0x40 [ 4492.302964] ? hrtimer_interrupt+0x128/0x2c0 [ 4492.306091] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4492.312594] ? apic_timer_interrupt+0xf/0x20 [ 4492.315476] [ 4492.316658] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4492.323893] do_raw_spin_lock+0x177/0x190 [ 4492.328854] _raw_spin_lock+0x21/0x30 [ 4492.331955] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4492.337442] ? do_raw_spin_unlock+0x75/0x190 [ 4492.339285] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4492.341592] ? do_wait_intr+0xf0/0xf0 [ 4492.343144] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4492.345223] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4492.347550] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4492.353942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4492.356482] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4492.360516] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4492.362911] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4492.365096] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4492.367708] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4492.373625] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4492.376473] lmv_intent_open+0x338/0x1150 [lmv] [ 4492.379131] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4492.382634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4492.385618] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4492.387749] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4492.390143] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4492.392949] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4492.400508] ? do_raw_spin_unlock+0x75/0x190 [ 4492.405200] ? _raw_spin_unlock+0x12/0x30 [ 4492.411478] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4492.415709] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4492.422543] ll_atomic_open+0x36a/0xd70 [lustre] [ 4492.430001] lookup_open+0x356/0xb40 [ 4492.432605] do_last+0x313/0xfc0 [ 4492.435186] ? nd_jump_root+0xe5/0x160 [ 4492.438839] ? path_init+0x437/0x520 [ 4492.443004] path_openat+0xf7/0x500 [ 4492.449364] ? do_raw_spin_unlock+0x75/0x190 [ 4492.451525] do_filp_open+0x99/0x140 [ 4492.457062] ? getname_flags+0x6e/0x330 [ 4492.458796] ? __check_object_size+0xff/0x256 [ 4492.465446] ? do_raw_spin_unlock+0x75/0x190 [ 4492.469060] ? _raw_spin_unlock+0x12/0x30 [ 4492.475028] do_sys_openat2+0x2b4/0x410 [ 4492.477676] do_sys_open+0x73/0xa0 [ 4492.482540] __x64_sys_openat+0x24/0x30 [ 4492.490933] do_syscall_64+0xc1/0x440 [ 4492.492735] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4492.501216] RIP: 0033:0x7f1a63983332 [ 4492.503562] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4492.525552] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4492.533661] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4492.541171] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4492.547810] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4492.553844] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4492.558155] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4498.400063] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2475s! [ 4498.404755] Showing busy workqueues and worker pools: [ 4498.407740] workqueue events: flags=0x0 [ 4498.410753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4498.414794] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4498.420426] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4498.424065] pending: sec_gc_main [ptlrpc] [ 4498.427903] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4498.432897] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4498.438865] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4498.442830] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4498.447281] workqueue events_long: flags=0x0 [ 4498.449857] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4498.453785] pending: lu_objects_flush [obdclass] [ 4498.457120] workqueue events_unbound: flags=0x2 [ 4498.459982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4498.463524] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4498.470166] workqueue events_power_efficient: flags=0x80 [ 4498.473389] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4498.476768] pending: neigh_periodic_work, do_cache_clean [ 4498.479868] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4498.483662] pending: neigh_periodic_work, check_lifetime [ 4498.486828] workqueue mm_percpu_wq: flags=0x8 [ 4498.489202] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4498.492775] pending: vmstat_update [ 4498.494860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4498.499113] pending: vmstat_update [ 4498.500865] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4498.504645] pending: vmstat_update [ 4498.506462] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4498.510079] pending: vmstat_update [ 4498.512498] workqueue writeback: flags=0x4a [ 4498.514910] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4498.518553] pending: wb_workfn, wb_workfn [ 4498.521253] workqueue xprtiod: flags=0xa [ 4498.523457] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4498.526709] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4498.530866] workqueue ipv6_addrconf: flags=0x40008 [ 4498.533754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4498.537305] pending: addrconf_verify_work [ 4498.539541] workqueue knbd0-recv: flags=0x1a [ 4498.541648] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4498.544399] in-flight: 647:recv_work [ 4498.546685] workqueue ptlrpc_pinger: flags=0x2 [ 4498.548801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4498.551835] pending: ptlrpc_pinger_main [ptlrpc] [ 4498.554944] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4488s workers=2 idle: 652 [ 4504.056409] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4504.060904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [ 4504.061362] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4504.061369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4504.061454] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4504.061460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4504.061463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4504.061477] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4504.061481] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4504.061485] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4504.061487] RDX: 00000000000066e7 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4504.061489] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4504.061491] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4504.061493] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4504.061496] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4504.061500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4504.061502] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4504.061511] Call Trace: [ 4504.061516] [ 4504.061521] ? show_regs.cold.9+0x22/0x2f [ 4504.061531] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4504.061538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4504.061543] ? watchdog+0x40/0x40 [ 4504.061546] ? hrtimer_interrupt+0x128/0x2c0 [ 4504.061551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4504.061558] ? apic_timer_interrupt+0xf/0x20 [ 4504.061563] [ 4504.061565] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4504.061570] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4504.061574] do_raw_spin_lock+0x177/0x190 [ 4504.061580] _raw_spin_lock+0x21/0x30 [ 4504.061585] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4504.061849] obd_mod_free+0x46/0x140 [ptlrpc] [ 4504.062470] mdc_commit_open+0x91/0xb0 [mdc] [ 4504.062510] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4504.075201] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4504.075342] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4504.075418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4504.075420] RIP: 0010:load_balance+0x725/0x1090 [ 4504.075500] Code: 48 83 00 01 9c 58 0f 1f 44 00 00 48 83 05 7a 6d dd 02 01 48 83 05 82 b9 dd 02 01 f6 c4 02 0f 85 22 03 00 00 48 8b 3c 24 57 9d <0f> 1f 44 00 00 8b 84 24 98 00 00 00 a8 02 74 23 83 e0 fd 48 83 05 [ 4504.075504] RSP: 0018:ffffc38540d24de0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4504.075509] RAX: 0000000000000006 RBX: ffff9f8452bbd030 RCX: ffff9f84820b3440 [ 4504.075511] RDX: 0000000000000008 RSI: ffff9f8452bbcf80 RDI: 0000000000000206 [ 4504.075514] RBP: ffffc38540d24e90 R08: ffff9f8450fba7c0 R09: 0000000000000000 [ 4504.075517] R10: ffff9f8452bbd018 R11: 000000000000a797 R12: ffff9f8452bbcf80 [ 4504.075519] R13: ffff9f8482033e08 R14: 0000000000000007 R15: ffff9f8482033380 [ 4504.075522] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4504.075524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4504.075526] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4504.075537] Call Trace: [ 4504.075543] [ 4504.075549] ? show_regs.cold.9+0x22/0x2f [ 4504.075559] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4504.087699] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4504.088715] after_reply+0xe2c/0x1780 [ptlrpc] [ 4504.089056] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4504.091608] ? __next_timer_interrupt+0x160/0x160 [ 4504.091621] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4504.091850] ? do_wait_intr+0xf0/0xf0 [ 4504.091858] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4504.093917] kthread+0x1d1/0x200 [ 4504.093925] ? set_kthread_struct+0x70/0x70 [ 4504.093941] ret_from_fork+0x1f/0x30 [ 4504.628262] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4504.630013] ? watchdog+0x40/0x40 [ 4504.634542] ? hrtimer_interrupt+0x128/0x2c0 [ 4504.636435] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4504.640524] ? apic_timer_interrupt+0xf/0x20 [ 4504.643786] ? apic_timer_interrupt+0xa/0x20 [ 4504.649862] ? load_balance+0x725/0x1090 [ 4504.651956] rebalance_domains+0x348/0x580 [ 4504.656060] ? kvm_clock_get_cycles+0x2c/0x50 [ 4504.663153] run_rebalance_domains+0x68/0xa0 [ 4504.666515] __do_softirq+0x134/0x4e8 [ 4504.669851] irq_exit_rcu+0x1c6/0x1e0 [ 4504.671943] irq_exit+0x12/0x30 [ 4504.674865] smp_apic_timer_interrupt+0xe0/0x270 [ 4504.677901] apic_timer_interrupt+0xf/0x20 [ 4504.680460] [ 4504.681324] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4504.684365] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4504.697905] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4504.703127] RAX: 0000000000006f81 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4504.708062] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4504.710963] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4504.714375] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4504.717322] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4504.720299] do_raw_spin_lock+0x177/0x190 [ 4504.721839] _raw_spin_lock+0x21/0x30 [ 4504.727552] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4504.734208] ? do_raw_spin_unlock+0x75/0x190 [ 4504.736457] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4504.739445] ? do_wait_intr+0xf0/0xf0 [ 4504.741078] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4504.743842] mdc_close+0x53b/0xd70 [mdc] [ 4504.745862] lmv_close+0x18d/0x490 [lmv] [ 4504.747761] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4504.750760] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4504.753387] ll_md_close+0x274/0x800 [lustre] [ 4504.755112] ll_file_release+0x128/0xc50 [lustre] [ 4504.757570] __fput+0x10d/0x440 [ 4504.759160] ____fput+0x12/0x20 [ 4504.760501] task_work_run+0xc6/0x110 [ 4504.762468] exit_to_usermode_loop+0x1dd/0x1f0 [ 4504.764699] do_syscall_64+0x3d6/0x440 [ 4504.766498] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4504.769403] RIP: 0033:0x7f1a63983cd8 [ 4504.771169] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4504.780550] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4504.786299] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4504.789745] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4504.792604] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4504.795646] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4504.798907] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4508.065936] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4508.087323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4508.159571] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4508.179575] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4508.192168] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4508.199665] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4508.237892] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4508.249377] RAX: 0000000000006d13 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4508.251959] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4508.262495] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4508.274259] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4508.289347] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4508.304664] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4508.331439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4508.340984] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4508.348946] Call Trace: [ 4508.360438] [ 4508.361229] ? show_regs.cold.9+0x22/0x2f [ 4508.362792] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4508.382816] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4508.396596] ? watchdog+0x40/0x40 [ 4508.404416] ? hrtimer_interrupt+0x128/0x2c0 [ 4508.416541] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4508.426359] ? apic_timer_interrupt+0xf/0x20 [ 4508.427999] [ 4508.428843] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4508.451013] do_raw_spin_lock+0x177/0x190 [ 4508.462428] _raw_spin_lock+0x21/0x30 [ 4508.463800] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4508.473042] ? do_raw_spin_unlock+0x75/0x190 [ 4508.477964] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4508.483148] ? do_wait_intr+0xf0/0xf0 [ 4508.487477] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4508.497074] mdc_close+0x53b/0xd70 [mdc] [ 4508.503523] lmv_close+0x18d/0x490 [lmv] [ 4508.508117] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4508.520841] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4508.530057] ll_md_close+0x274/0x800 [lustre] [ 4508.543268] ll_file_release+0x128/0xc50 [lustre] [ 4508.557084] __fput+0x10d/0x440 [ 4508.560634] ____fput+0x12/0x20 [ 4508.569155] task_work_run+0xc6/0x110 [ 4508.570621] exit_to_usermode_loop+0x1dd/0x1f0 [ 4508.586181] do_syscall_64+0x3d6/0x440 [ 4508.592620] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4508.608422] RIP: 0033:0x7f1a63983cd8 [ 4508.613614] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4508.646548] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4508.660732] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4508.673546] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4508.686770] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4508.704672] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4508.725948] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4524.054842] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4524.067129] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4524.121796] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4524.131572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4524.140363] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4524.143403] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4524.159452] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4524.163971] RAX: 00000000000018f2 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4524.169081] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4524.172869] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4524.177033] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4524.181389] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4524.184612] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4524.190031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4524.194588] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4524.198291] Call Trace: [ 4524.199853] [ 4524.201552] ? show_regs.cold.9+0x22/0x2f [ 4524.203324] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4524.207163] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4524.208974] ? watchdog+0x40/0x40 [ 4524.210973] ? hrtimer_interrupt+0x128/0x2c0 [ 4524.215599] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4524.218238] ? apic_timer_interrupt+0xf/0x20 [ 4524.220874] [ 4524.221942] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4524.226925] do_raw_spin_lock+0x177/0x190 [ 4524.228881] _raw_spin_lock+0x21/0x30 [ 4524.231626] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4524.236298] ? do_raw_spin_unlock+0x75/0x190 [ 4524.241769] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4524.246655] ? do_wait_intr+0xf0/0xf0 [ 4524.249228] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4524.255648] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4524.259421] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4524.266336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.275300] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4524.280473] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4524.283005] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4524.287226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.292609] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4524.300657] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4524.311651] lmv_intent_open+0x338/0x1150 [lmv] [ 4524.317509] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.322931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.330194] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4524.336520] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4524.346549] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.353781] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4524.356478] ? do_raw_spin_unlock+0x75/0x190 [ 4524.363531] ? _raw_spin_unlock+0x12/0x30 [ 4524.374151] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4524.381221] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4524.383040] ll_atomic_open+0x36a/0xd70 [lustre] [ 4524.393107] lookup_open+0x356/0xb40 [ 4524.395393] do_last+0x313/0xfc0 [ 4524.397022] ? nd_jump_root+0xe5/0x160 [ 4524.399153] ? path_init+0x437/0x520 [ 4524.401551] path_openat+0xf7/0x500 [ 4524.403650] ? do_raw_spin_unlock+0x75/0x190 [ 4524.407721] do_filp_open+0x99/0x140 [ 4524.413335] ? getname_flags+0x6e/0x330 [ 4524.419506] ? __check_object_size+0xff/0x256 [ 4524.421959] ? do_raw_spin_unlock+0x75/0x190 [ 4524.424752] ? _raw_spin_unlock+0x12/0x30 [ 4524.428743] do_sys_openat2+0x2b4/0x410 [ 4524.432492] do_sys_open+0x73/0xa0 [ 4524.435682] __x64_sys_openat+0x24/0x30 [ 4524.437901] do_syscall_64+0xc1/0x440 [ 4524.445314] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4524.448832] RIP: 0033:0x7f1a63983332 [ 4524.456290] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4524.474590] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4524.477618] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4524.480093] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4524.484184] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4524.495497] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4524.500352] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4528.612059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2505s! [ 4528.616292] Showing busy workqueues and worker pools: [ 4528.619675] workqueue events: flags=0x0 [ 4528.623496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4528.628447] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4528.632872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4528.635791] pending: sec_gc_main [ptlrpc] [ 4528.638515] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4528.641718] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4528.656090] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4528.669683] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4528.681068] workqueue events_long: flags=0x0 [ 4528.699582] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4528.708939] pending: lu_objects_flush [obdclass] [ 4528.711025] workqueue events_unbound: flags=0x2 [ 4528.716907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4528.725602] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4528.741969] workqueue events_power_efficient: flags=0x80 [ 4528.751179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4528.761781] pending: neigh_periodic_work, do_cache_clean [ 4528.781354] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4528.791050] pending: neigh_periodic_work, check_lifetime [ 4528.800629] workqueue mm_percpu_wq: flags=0x8 [ 4528.811351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4528.820890] pending: vmstat_update [ 4528.822736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4528.825402] pending: vmstat_update [ 4528.839929] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4528.857148] pending: vmstat_update [ 4528.864900] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4528.876691] pending: vmstat_update [ 4528.888813] workqueue writeback: flags=0x4a [ 4528.896751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4528.904696] pending: wb_workfn, wb_workfn [ 4528.913262] workqueue xprtiod: flags=0xa [ 4528.927849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4528.933230] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4528.936837] workqueue ipv6_addrconf: flags=0x40008 [ 4528.938721] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4528.946245] pending: addrconf_verify_work [ 4528.955221] workqueue knbd0-recv: flags=0x1a [ 4528.960842] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4528.965771] in-flight: 647:recv_work [ 4528.967766] workqueue ptlrpc_pinger: flags=0x2 [ 4528.969718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4528.974573] pending: ptlrpc_pinger_main [ptlrpc] [ 4528.983198] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4519s workers=2 idle: 652 [ 4532.058736] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4532.058962] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4532.075420] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4532.076082] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4532.076089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4532.076092] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4532.076107] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4532.076112] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4532.076116] RAX: 0000000000006909 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4532.076121] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4532.076123] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4532.076125] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4532.076128] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4532.076131] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4532.076134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4532.076137] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4532.076144] Call Trace: [ 4532.076154] [ 4532.076160] ? show_regs.cold.9+0x22/0x2f [ 4532.076170] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4532.076177] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4532.076181] ? watchdog+0x40/0x40 [ 4532.076185] ? hrtimer_interrupt+0x128/0x2c0 [ 4532.076189] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4532.076197] ? apic_timer_interrupt+0xf/0x20 [ 4532.076202] [ 4532.076204] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4532.076210] do_raw_spin_lock+0x177/0x190 [ 4532.076215] _raw_spin_lock+0x21/0x30 [ 4532.076220] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4532.076934] ? do_raw_spin_unlock+0x75/0x190 [ 4532.076940] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4532.077327] ? do_wait_intr+0xf0/0xf0 [ 4532.077335] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4532.077562] mdc_close+0x53b/0xd70 [mdc] [ 4532.077617] lmv_close+0x18d/0x490 [lmv] [ 4532.077659] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4532.077937] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4532.078084] ll_md_close+0x274/0x800 [lustre] [ 4532.078257] ll_file_release+0x128/0xc50 [lustre] [ 4532.078403] __fput+0x10d/0x440 [ 4532.078416] ____fput+0x12/0x20 [ 4532.078420] task_work_run+0xc6/0x110 [ 4532.078429] exit_to_usermode_loop+0x1dd/0x1f0 [ 4532.078436] do_syscall_64+0x3d6/0x440 [ 4532.078441] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4532.078449] RIP: 0033:0x7f1a63983cd8 [ 4532.078457] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4532.078460] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4532.078465] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4532.078467] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4532.078469] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4532.078471] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4532.078473] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4532.339469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4532.365346] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4532.380495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4532.385820] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4532.388181] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4532.411400] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4532.421037] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4532.427471] RDX: 000000000000379b RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4532.431157] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4532.438288] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4532.443449] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4532.456258] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4532.461751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4532.466076] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4532.471844] Call Trace: [ 4532.472790] [ 4532.473537] ? show_regs.cold.9+0x22/0x2f [ 4532.480523] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4532.483464] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4532.487365] ? watchdog+0x40/0x40 [ 4532.493213] ? hrtimer_interrupt+0x128/0x2c0 [ 4532.497642] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4532.500616] ? apic_timer_interrupt+0xf/0x20 [ 4532.502639] [ 4532.503858] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4532.506609] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4532.510063] do_raw_spin_lock+0x177/0x190 [ 4532.512558] _raw_spin_lock+0x21/0x30 [ 4532.514579] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4532.517755] obd_mod_free+0x46/0x140 [ptlrpc] [ 4532.520320] mdc_commit_open+0x91/0xb0 [mdc] [ 4532.522385] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4532.525155] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4532.527928] after_reply+0xe2c/0x1780 [ptlrpc] [ 4532.530767] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4532.533516] ? __next_timer_interrupt+0x160/0x160 [ 4532.535614] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4532.537875] ? do_wait_intr+0xf0/0xf0 [ 4532.539494] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4532.542128] kthread+0x1d1/0x200 [ 4532.543603] ? set_kthread_struct+0x70/0x70 [ 4532.545446] ret_from_fork+0x1f/0x30 [ 4536.057021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4536.060286] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4536.086161] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4536.089503] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4536.092666] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4536.094175] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4536.102131] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4536.104551] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4536.106970] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4536.109116] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4536.111558] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4536.114102] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4536.116659] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4536.120179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4536.122210] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4536.125187] Call Trace: [ 4536.126167] [ 4536.126712] ? show_regs.cold.9+0x22/0x2f [ 4536.128523] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4536.130317] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4536.132397] ? watchdog+0x40/0x40 [ 4536.133673] ? hrtimer_interrupt+0x128/0x2c0 [ 4536.135400] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4536.137553] ? apic_timer_interrupt+0xf/0x20 [ 4536.139566] [ 4536.140589] ? native_safe_halt+0x2e/0x50 [ 4536.142627] kvm_wait+0xcd/0xe0 [ 4536.144163] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4536.148759] do_raw_spin_lock+0x177/0x190 [ 4536.151485] _raw_spin_lock+0x21/0x30 [ 4536.156622] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4536.164513] ? do_raw_spin_unlock+0x75/0x190 [ 4536.167678] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4536.172996] ? do_wait_intr+0xf0/0xf0 [ 4536.176506] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4536.179382] mdc_close+0x53b/0xd70 [mdc] [ 4536.180456] lmv_close+0x18d/0x490 [lmv] [ 4536.187634] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4536.190695] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4536.192868] ll_md_close+0x274/0x800 [lustre] [ 4536.196138] ll_file_release+0x128/0xc50 [lustre] [ 4536.198603] __fput+0x10d/0x440 [ 4536.200275] ____fput+0x12/0x20 [ 4536.201649] task_work_run+0xc6/0x110 [ 4536.203587] exit_to_usermode_loop+0x1dd/0x1f0 [ 4536.206402] do_syscall_64+0x3d6/0x440 [ 4536.208229] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4536.210478] RIP: 0033:0x7f1a63983cd8 [ 4536.212325] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4536.221298] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4536.224347] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4536.227610] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4536.230423] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4536.233215] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4536.236359] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4556.053018] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 4556.058601] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4556.096375] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4556.107349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4556.121905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4556.129393] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4556.148165] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4556.157571] RAX: 0000000000005790 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4556.166888] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4556.176037] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4556.181575] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4556.197920] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4556.202026] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4556.207271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4556.212361] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4556.221813] Call Trace: [ 4556.224270] [ 4556.227478] ? show_regs.cold.9+0x22/0x2f [ 4556.231780] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4556.234570] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4556.236604] ? watchdog+0x40/0x40 [ 4556.243434] ? hrtimer_interrupt+0x128/0x2c0 [ 4556.247006] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4556.252900] ? apic_timer_interrupt+0xf/0x20 [ 4556.257941] [ 4556.259245] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4556.264857] do_raw_spin_lock+0x177/0x190 [ 4556.269401] _raw_spin_lock+0x21/0x30 [ 4556.277613] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4556.285978] ? do_raw_spin_unlock+0x75/0x190 [ 4556.288348] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4556.292172] ? do_wait_intr+0xf0/0xf0 [ 4556.295440] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4556.299574] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4556.305945] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4556.314040] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4556.320460] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4556.323917] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4556.329526] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4556.335541] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4556.341186] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4556.352251] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4556.359993] lmv_intent_open+0x338/0x1150 [lmv] [ 4556.366542] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4556.371002] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4556.374450] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4556.380539] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4556.385726] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4556.391916] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4556.394400] ? do_raw_spin_unlock+0x75/0x190 [ 4556.396717] ? _raw_spin_unlock+0x12/0x30 [ 4556.399938] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4556.402961] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4556.409043] ll_atomic_open+0x36a/0xd70 [lustre] [ 4556.415639] lookup_open+0x356/0xb40 [ 4556.417967] do_last+0x313/0xfc0 [ 4556.423947] ? nd_jump_root+0xe5/0x160 [ 4556.430443] ? path_init+0x437/0x520 [ 4556.436817] path_openat+0xf7/0x500 [ 4556.441820] ? do_raw_spin_unlock+0x75/0x190 [ 4556.450369] do_filp_open+0x99/0x140 [ 4556.458469] ? getname_flags+0x6e/0x330 [ 4556.462161] ? __check_object_size+0xff/0x256 [ 4556.467987] ? do_raw_spin_unlock+0x75/0x190 [ 4556.477192] ? _raw_spin_unlock+0x12/0x30 [ 4556.481470] do_sys_openat2+0x2b4/0x410 [ 4556.485892] do_sys_open+0x73/0xa0 [ 4556.488065] __x64_sys_openat+0x24/0x30 [ 4556.490178] do_syscall_64+0xc1/0x440 [ 4556.491890] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4556.494226] RIP: 0033:0x7f1a63983332 [ 4556.495672] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4556.502449] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4556.505657] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4556.507566] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4556.510461] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4556.512951] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4556.515569] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4559.330850] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2536s! [ 4559.333891] Showing busy workqueues and worker pools: [ 4559.335393] workqueue events: flags=0x0 [ 4559.340115] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4559.342143] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4559.351569] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4559.355665] pending: sec_gc_main [ptlrpc] [ 4559.366859] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4559.372971] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4559.378515] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4559.381560] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4559.389217] workqueue events_long: flags=0x0 [ 4559.390627] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4559.395860] pending: lu_objects_flush [obdclass] [ 4559.400598] workqueue events_unbound: flags=0x2 [ 4559.404109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4559.406855] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4559.415846] workqueue events_power_efficient: flags=0x80 [ 4559.419694] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4559.423301] pending: neigh_periodic_work, do_cache_clean [ 4559.426473] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4559.430358] pending: neigh_periodic_work, check_lifetime [ 4559.433206] workqueue mm_percpu_wq: flags=0x8 [ 4559.435571] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4559.439100] pending: vmstat_update [ 4559.441018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4559.444422] pending: vmstat_update [ 4559.446299] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4559.449671] pending: vmstat_update [ 4559.451597] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4559.455282] pending: vmstat_update [ 4559.456922] workqueue writeback: flags=0x4a [ 4559.458796] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4559.461526] pending: wb_workfn, wb_workfn [ 4559.463669] workqueue xprtiod: flags=0xa [ 4559.465631] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4559.469269] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4559.472917] workqueue ipv6_addrconf: flags=0x40008 [ 4559.475064] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4559.477885] pending: addrconf_verify_work [ 4559.480262] workqueue knbd0-recv: flags=0x1a [ 4559.482042] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4559.485063] in-flight: 647:recv_work [ 4559.487044] workqueue ptlrpc_pinger: flags=0x2 [ 4559.489016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4559.491782] pending: ptlrpc_pinger_main [ptlrpc] [ 4559.494294] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4549s workers=2 idle: 652 [ 4560.056062] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4560.058839] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4560.059769] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4560.091897] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4560.091909] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4560.091911] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4560.091928] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4560.091932] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4560.091938] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4560.091940] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4560.091942] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4560.091944] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4560.091947] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4560.091950] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4560.091953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4560.091956] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4560.091964] Call Trace: [ 4560.091970] [ 4560.092031] ? show_regs.cold.9+0x22/0x2f [ 4560.092042] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4560.092048] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4560.092053] ? watchdog+0x40/0x40 [ 4560.092057] ? hrtimer_interrupt+0x128/0x2c0 [ 4560.092062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4560.092068] ? apic_timer_interrupt+0xf/0x20 [ 4560.092073] [ 4560.092076] ? native_safe_halt+0x2e/0x50 [ 4560.092081] kvm_wait+0xcd/0xe0 [ 4560.092088] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4560.092096] do_raw_spin_lock+0x177/0x190 [ 4560.092101] _raw_spin_lock+0x21/0x30 [ 4560.092105] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4560.092418] ? do_raw_spin_unlock+0x75/0x190 [ 4560.092424] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4560.092690] ? do_wait_intr+0xf0/0xf0 [ 4560.092708] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4560.093052] mdc_close+0x53b/0xd70 [mdc] [ 4560.093091] lmv_close+0x18d/0x490 [lmv] [ 4560.093122] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4560.093230] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4560.093307] ll_md_close+0x274/0x800 [lustre] [ 4560.093389] ll_file_release+0x128/0xc50 [lustre] [ 4560.093474] __fput+0x10d/0x440 [ 4560.094073] ____fput+0x12/0x20 [ 4560.094080] task_work_run+0xc6/0x110 [ 4560.094087] exit_to_usermode_loop+0x1dd/0x1f0 [ 4560.094093] do_syscall_64+0x3d6/0x440 [ 4560.094098] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4560.094106] RIP: 0033:0x7f1a63983cd8 [ 4560.094114] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4560.094121] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4560.094126] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4560.094128] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4560.094130] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4560.094132] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4560.094134] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4560.291507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4560.323489] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4560.329201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4560.333317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4560.335662] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4560.343367] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4560.346276] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4560.348391] RDX: 0000000000007f81 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4560.350645] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4560.352886] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4560.355118] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4560.357330] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4560.359971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4560.361789] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4560.363863] Call Trace: [ 4560.364918] [ 4560.365477] ? show_regs.cold.9+0x22/0x2f [ 4560.366748] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4560.368247] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4560.369667] ? watchdog+0x40/0x40 [ 4560.370797] ? hrtimer_interrupt+0x128/0x2c0 [ 4560.372150] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4560.373670] ? apic_timer_interrupt+0xf/0x20 [ 4560.375336] [ 4560.376122] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4560.377864] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4560.379431] do_raw_spin_lock+0x177/0x190 [ 4560.380879] _raw_spin_lock+0x21/0x30 [ 4560.382268] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4560.385305] obd_mod_free+0x46/0x140 [ptlrpc] [ 4560.389924] mdc_commit_open+0x91/0xb0 [mdc] [ 4560.392951] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4560.395491] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4560.398208] after_reply+0xe2c/0x1780 [ptlrpc] [ 4560.400157] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4560.402445] ? __next_timer_interrupt+0x160/0x160 [ 4560.404298] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4560.406158] ? do_wait_intr+0xf0/0xf0 [ 4560.407301] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4560.409334] kthread+0x1d1/0x200 [ 4560.410426] ? set_kthread_struct+0x70/0x70 [ 4560.411898] ret_from_fork+0x1f/0x30 [ 4564.059248] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4564.081972] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4564.148229] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4564.154431] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4564.159392] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4564.162383] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4564.170570] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4564.174624] RAX: 0000000000000386 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4564.178483] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4564.183112] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4564.187254] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4564.190143] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4564.193087] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4564.196525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4564.199255] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4564.203978] Call Trace: [ 4564.205027] [ 4564.205842] ? show_regs.cold.9+0x22/0x2f [ 4564.207579] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4564.212211] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4564.214125] ? watchdog+0x40/0x40 [ 4564.215450] ? hrtimer_interrupt+0x128/0x2c0 [ 4564.217747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4564.220312] ? apic_timer_interrupt+0xf/0x20 [ 4564.222255] [ 4564.223282] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4564.225468] do_raw_spin_lock+0x177/0x190 [ 4564.227456] _raw_spin_lock+0x21/0x30 [ 4564.229099] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4564.231523] ? do_raw_spin_unlock+0x75/0x190 [ 4564.233288] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4564.236239] ? do_wait_intr+0xf0/0xf0 [ 4564.238642] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4564.241605] mdc_close+0x53b/0xd70 [mdc] [ 4564.243756] lmv_close+0x18d/0x490 [lmv] [ 4564.246296] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4564.248893] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4564.251585] ll_md_close+0x274/0x800 [lustre] [ 4564.254638] ll_file_release+0x128/0xc50 [lustre] [ 4564.256629] __fput+0x10d/0x440 [ 4564.258701] ____fput+0x12/0x20 [ 4564.262668] task_work_run+0xc6/0x110 [ 4564.265277] exit_to_usermode_loop+0x1dd/0x1f0 [ 4564.272853] do_syscall_64+0x3d6/0x440 [ 4564.274455] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4564.278858] RIP: 0033:0x7f1a63983cd8 [ 4564.280720] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4564.303350] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4564.317629] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4564.332373] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4564.340023] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4564.352415] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4564.362631] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4584.059788] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 4584.062801] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4584.082195] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4584.086531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4584.090082] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4584.092264] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4584.098931] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4584.101508] RAX: 00000000000052c5 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4584.104073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4584.106422] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4584.110152] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4584.115012] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4584.118853] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4584.123053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4584.126121] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4584.131134] Call Trace: [ 4584.132589] [ 4584.133340] ? show_regs.cold.9+0x22/0x2f [ 4584.137112] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4584.141213] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4584.145232] ? watchdog+0x40/0x40 [ 4584.147172] ? hrtimer_interrupt+0x128/0x2c0 [ 4584.155321] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4584.161885] ? apic_timer_interrupt+0xf/0x20 [ 4584.164848] [ 4584.165942] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4584.170048] do_raw_spin_lock+0x177/0x190 [ 4584.172709] _raw_spin_lock+0x21/0x30 [ 4584.175048] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4584.177678] ? do_raw_spin_unlock+0x75/0x190 [ 4584.179902] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4584.183152] ? do_wait_intr+0xf0/0xf0 [ 4584.185707] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4584.188951] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4584.192082] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4584.195064] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4584.197853] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4584.201045] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4584.206691] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4584.210689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4584.214156] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4584.217446] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4584.225682] lmv_intent_open+0x338/0x1150 [lmv] [ 4584.228813] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4584.234031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4584.237435] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4584.240453] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4584.243844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4584.247965] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4584.251708] ? do_raw_spin_unlock+0x75/0x190 [ 4584.254887] ? _raw_spin_unlock+0x12/0x30 [ 4584.258101] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4584.260857] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4584.268678] ll_atomic_open+0x36a/0xd70 [lustre] [ 4584.270089] lookup_open+0x356/0xb40 [ 4584.277983] do_last+0x313/0xfc0 [ 4584.281899] ? nd_jump_root+0xe5/0x160 [ 4584.285004] ? path_init+0x437/0x520 [ 4584.286921] path_openat+0xf7/0x500 [ 4584.289409] ? do_raw_spin_unlock+0x75/0x190 [ 4584.292281] do_filp_open+0x99/0x140 [ 4584.294153] ? getname_flags+0x6e/0x330 [ 4584.296543] ? __check_object_size+0xff/0x256 [ 4584.299126] ? do_raw_spin_unlock+0x75/0x190 [ 4584.301568] ? _raw_spin_unlock+0x12/0x30 [ 4584.303972] do_sys_openat2+0x2b4/0x410 [ 4584.306658] do_sys_open+0x73/0xa0 [ 4584.308138] __x64_sys_openat+0x24/0x30 [ 4584.309922] do_syscall_64+0xc1/0x440 [ 4584.312819] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4584.315898] RIP: 0033:0x7f1a63983332 [ 4584.319705] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4584.339235] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4584.349176] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4584.357544] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4584.363503] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4584.370450] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4584.377168] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4588.056120] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4588.063709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4588.064826] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4588.091820] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4588.091887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4588.091891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4588.091906] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4588.091909] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4588.091915] RAX: 0000000000003512 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4588.091917] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4588.091919] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4588.091920] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4588.091922] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4588.091926] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4588.091928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4588.091930] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4588.091939] Call Trace: [ 4588.091945] [ 4588.091951] ? show_regs.cold.9+0x22/0x2f [ 4588.091961] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4588.091967] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4588.091972] ? watchdog+0x40/0x40 [ 4588.091976] ? hrtimer_interrupt+0x128/0x2c0 [ 4588.091979] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4588.091987] ? apic_timer_interrupt+0xf/0x20 [ 4588.091991] [ 4588.091994] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4588.092001] do_raw_spin_lock+0x177/0x190 [ 4588.092006] _raw_spin_lock+0x21/0x30 [ 4588.092011] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4588.092555] ? do_raw_spin_unlock+0x75/0x190 [ 4588.092561] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4588.092762] ? do_wait_intr+0xf0/0xf0 [ 4588.092770] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4588.093122] mdc_close+0x53b/0xd70 [mdc] [ 4588.093162] lmv_close+0x18d/0x490 [lmv] [ 4588.093195] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4588.093308] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4588.093560] ll_md_close+0x274/0x800 [lustre] [ 4588.093641] ll_file_release+0x128/0xc50 [lustre] [ 4588.093729] __fput+0x10d/0x440 [ 4588.093737] ____fput+0x12/0x20 [ 4588.093742] task_work_run+0xc6/0x110 [ 4588.093748] exit_to_usermode_loop+0x1dd/0x1f0 [ 4588.093755] do_syscall_64+0x3d6/0x440 [ 4588.093760] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4588.093767] RIP: 0033:0x7f1a63983cd8 [ 4588.093774] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4588.093778] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4588.093783] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4588.093785] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4588.093787] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4588.093789] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4588.093791] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4588.393833] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4588.420057] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4588.427826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4588.433015] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4588.435720] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4588.455495] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4588.462072] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4588.469045] RDX: 00000000000067a2 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4588.477741] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4588.485230] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4588.492904] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4588.497471] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4588.505833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4588.512402] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4588.519441] Call Trace: [ 4588.520980] [ 4588.521865] ? show_regs.cold.9+0x22/0x2f [ 4588.525310] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4588.527111] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4588.530919] ? watchdog+0x40/0x40 [ 4588.532563] ? hrtimer_interrupt+0x128/0x2c0 [ 4588.535137] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4588.539918] ? apic_timer_interrupt+0xf/0x20 [ 4588.543403] [ 4588.546790] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4588.553689] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4588.560018] do_raw_spin_lock+0x177/0x190 [ 4588.564851] _raw_spin_lock+0x21/0x30 [ 4588.568201] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4588.572953] obd_mod_free+0x46/0x140 [ptlrpc] [ 4588.575266] mdc_commit_open+0x91/0xb0 [mdc] [ 4588.579342] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4588.582927] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4588.587308] after_reply+0xe2c/0x1780 [ptlrpc] [ 4588.591852] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4588.596985] ? __next_timer_interrupt+0x160/0x160 [ 4588.600146] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4588.605311] ? do_wait_intr+0xf0/0xf0 [ 4588.608294] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4588.615570] kthread+0x1d1/0x200 [ 4588.616711] ? set_kthread_struct+0x70/0x70 [ 4588.618213] ret_from_fork+0x1f/0x30 [ 4589.542599] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2566s! [ 4589.549498] Showing busy workqueues and worker pools: [ 4589.553655] workqueue events: flags=0x0 [ 4589.564293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4589.577289] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4589.587052] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4589.599748] pending: sec_gc_main [ptlrpc] [ 4589.609569] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4589.617826] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4589.640051] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4589.652557] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4589.662192] workqueue events_long: flags=0x0 [ 4589.673915] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4589.676326] pending: lu_objects_flush [obdclass] [ 4589.685811] workqueue events_unbound: flags=0x2 [ 4589.690012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4589.692330] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4589.699796] workqueue events_power_efficient: flags=0x80 [ 4589.710803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4589.714081] pending: neigh_periodic_work, do_cache_clean [ 4589.724569] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4589.730987] pending: neigh_periodic_work, check_lifetime [ 4589.734032] workqueue mm_percpu_wq: flags=0x8 [ 4589.736369] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4589.744263] pending: vmstat_update [ 4589.749824] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4589.755607] pending: vmstat_update [ 4589.760805] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4589.765719] pending: vmstat_update [ 4589.773702] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4589.777597] pending: vmstat_update [ 4589.780019] workqueue writeback: flags=0x4a [ 4589.783758] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4589.788234] pending: wb_workfn, wb_workfn [ 4589.794616] workqueue xprtiod: flags=0xa [ 4589.799768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4589.805467] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4589.810623] workqueue ipv6_addrconf: flags=0x40008 [ 4589.818833] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4589.825984] pending: addrconf_verify_work [ 4589.831519] workqueue knbd0-recv: flags=0x1a [ 4589.833562] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4589.839198] in-flight: 647:recv_work [ 4589.841041] workqueue ptlrpc_pinger: flags=0x2 [ 4589.843145] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4589.846183] pending: ptlrpc_pinger_main [ptlrpc] [ 4589.849094] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4579s workers=2 idle: 652 [ 4592.057107] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4592.066841] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4592.095029] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4592.102282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4592.107595] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4592.110710] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4592.120225] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4592.124766] RAX: 0000000000005858 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4592.129364] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4592.133742] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4592.137704] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4592.141546] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4592.144611] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4592.153201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4592.158515] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4592.164727] Call Trace: [ 4592.165780] [ 4592.168503] ? show_regs.cold.9+0x22/0x2f [ 4592.170471] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4592.173046] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4592.175179] ? watchdog+0x40/0x40 [ 4592.177442] ? hrtimer_interrupt+0x128/0x2c0 [ 4592.179608] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4592.182197] ? apic_timer_interrupt+0xf/0x20 [ 4592.184313] [ 4592.185375] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4592.187979] do_raw_spin_lock+0x177/0x190 [ 4592.190168] _raw_spin_lock+0x21/0x30 [ 4592.191985] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4592.195402] ? do_raw_spin_unlock+0x75/0x190 [ 4592.197400] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4592.200580] ? do_wait_intr+0xf0/0xf0 [ 4592.202606] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4592.205690] mdc_close+0x53b/0xd70 [mdc] [ 4592.208190] lmv_close+0x18d/0x490 [lmv] [ 4592.210270] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4592.213721] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4592.217312] ll_md_close+0x274/0x800 [lustre] [ 4592.222174] ll_file_release+0x128/0xc50 [lustre] [ 4592.226220] __fput+0x10d/0x440 [ 4592.229649] ____fput+0x12/0x20 [ 4592.233931] task_work_run+0xc6/0x110 [ 4592.240100] exit_to_usermode_loop+0x1dd/0x1f0 [ 4592.242669] do_syscall_64+0x3d6/0x440 [ 4592.245726] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4592.250906] RIP: 0033:0x7f1a63983cd8 [ 4592.254989] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4592.271046] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4592.275337] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4592.280054] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4592.285343] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4592.290183] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4592.294959] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4602.469746] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4602.473539] rcu: 3-...!: (1554350 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 4602.491457] (detected by 1, t=2580063 jiffies, g=269633, q=1038) [ 4602.495552] Sending NMI from CPU 1 to CPUs 3: [ 4602.502899] NMI backtrace for cpu 3 [ 4602.502906] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4602.502908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4602.502909] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4602.502911] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4602.502913] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 4602.502915] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4602.502917] RDX: 0000000000007a7e RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4602.502918] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4602.502919] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4602.502920] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4602.502921] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4602.502923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4602.502924] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4602.502925] Call Trace: [ 4602.502926] [ 4602.502926] ? show_regs.cold.9+0x22/0x2f [ 4602.502927] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4602.502928] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4602.502929] ? nmi_handle+0xbd/0x210 [ 4602.502929] ? default_do_nmi+0x4b/0x1b0 [ 4602.502930] ? do_nmi+0x280/0x380 [ 4602.502931] ? end_repeat_nmi+0x16/0x1f [ 4602.502932] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4602.502933] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4602.502933] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4602.502935] [ 4602.502935] do_raw_spin_lock+0x177/0x190 [ 4602.502936] _raw_spin_lock+0x21/0x30 [ 4602.502937] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4602.502938] obd_mod_free+0x46/0x140 [ptlrpc] [ 4602.502939] mdc_commit_open+0x91/0xb0 [mdc] [ 4602.502940] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4602.502941] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4602.502941] after_reply+0xe2c/0x1780 [ptlrpc] [ 4602.502942] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4602.502943] ? __next_timer_interrupt+0x160/0x160 [ 4602.502944] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4602.502944] ? do_wait_intr+0xf0/0xf0 [ 4602.502945] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4602.502946] kthread+0x1d1/0x200 [ 4602.502947] ? set_kthread_struct+0x70/0x70 [ 4602.502948] ret_from_fork+0x1f/0x30 [ 4602.507337] rcu: rcu_sched kthread starved for 2579709 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 4602.729381] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4602.732120] rcu: RCU grace-period kthread stack dump: [ 4602.734390] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4602.739989] Call Trace: [ 4602.741213] __schedule+0x351/0xcb0 [ 4602.743152] schedule+0xc0/0x180 [ 4602.744896] schedule_timeout+0xb4/0x190 [ 4602.747126] ? do_raw_spin_unlock+0x75/0x190 [ 4602.751598] ? __next_timer_interrupt+0x160/0x160 [ 4602.755078] ? prepare_to_swait_event+0xf3/0x140 [ 4602.756300] rcu_gp_kthread+0x8dd/0x15b0 [ 4602.762762] ? force_qs_rnp+0x2c0/0x2c0 [ 4602.766603] kthread+0x1d1/0x200 [ 4602.768016] ? set_kthread_struct+0x70/0x70 [ 4602.773228] ret_from_fork+0x1f/0x30 [ 4602.774382] rcu: Stack dump where RCU GP kthread last ran: [ 4602.778978] NMI backtrace for cpu 1 [ 4602.781597] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4602.786997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4602.793833] Call Trace: [ 4602.796086] [ 4602.796864] ? dump_stack+0xbb/0x10e [ 4602.801250] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4602.808983] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4602.810702] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4602.814112] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4602.816711] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4602.818694] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 4602.824689] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 4602.826676] ? update_process_times+0x6d/0xc0 [ 4602.828107] ? tick_sched_handle.isra.18+0x27/0x90 [ 4602.831904] ? tick_sched_timer+0x53/0xb0 [ 4602.833133] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4602.834979] ? tick_sched_handle.isra.18+0x90/0x90 [ 4602.841156] ? hrtimer_interrupt+0x128/0x2c0 [ 4602.844250] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4602.846620] ? apic_timer_interrupt+0xf/0x20 [ 4602.848787] [ 4602.849526] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4602.851838] ? do_raw_spin_lock+0x177/0x190 [ 4602.853269] ? _raw_spin_lock+0x21/0x30 [ 4602.856212] ? ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4602.861884] ? do_raw_spin_unlock+0x75/0x190 [ 4602.864594] ? ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4602.870054] ? do_wait_intr+0xf0/0xf0 [ 4602.873492] ? ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4602.875967] ? mdc_close+0x53b/0xd70 [mdc] [ 4602.879706] ? lmv_close+0x18d/0x490 [lmv] [ 4602.880806] ? ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4602.886623] ? ll_md_real_close+0xa5/0x2a0 [lustre] [ 4602.891837] ? ll_md_close+0x274/0x800 [lustre] [ 4602.896998] ? ll_file_release+0x128/0xc50 [lustre] [ 4602.899642] ? __fput+0x10d/0x440 [ 4602.901186] ? ____fput+0x12/0x20 [ 4602.904393] ? task_work_run+0xc6/0x110 [ 4602.905907] ? exit_to_usermode_loop+0x1dd/0x1f0 [ 4602.908380] ? do_syscall_64+0x3d6/0x440 [ 4602.909937] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4616.055873] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 4616.070334] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4616.130185] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4616.147126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4616.156839] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4616.161194] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4616.178557] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4616.190127] RAX: 0000000000007c86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4616.198471] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4616.201999] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4616.209944] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4616.216381] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4616.218471] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4616.229298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4616.232290] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4616.238948] Call Trace: [ 4616.240121] [ 4616.244366] ? show_regs.cold.9+0x22/0x2f [ 4616.252493] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4616.254392] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4616.259626] ? watchdog+0x40/0x40 [ 4616.263770] ? hrtimer_interrupt+0x128/0x2c0 [ 4616.271806] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4616.276242] ? apic_timer_interrupt+0xf/0x20 [ 4616.281018] [ 4616.282561] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4616.288046] do_raw_spin_lock+0x177/0x190 [ 4616.291511] _raw_spin_lock+0x21/0x30 [ 4616.296201] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4616.313656] ? do_raw_spin_unlock+0x75/0x190 [ 4616.323199] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4616.328440] ? do_wait_intr+0xf0/0xf0 [ 4616.334674] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4616.348602] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4616.355749] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4616.363745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4616.370299] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4616.376430] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4616.377772] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4616.383720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4616.391644] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4616.402261] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4616.410548] lmv_intent_open+0x338/0x1150 [lmv] [ 4616.412455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4616.418718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4616.422082] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4616.425355] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4616.429118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4616.431767] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4616.433695] ? do_raw_spin_unlock+0x75/0x190 [ 4616.435724] ? _raw_spin_unlock+0x12/0x30 [ 4616.438055] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4616.440565] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4616.442978] ll_atomic_open+0x36a/0xd70 [lustre] [ 4616.445441] lookup_open+0x356/0xb40 [ 4616.451121] do_last+0x313/0xfc0 [ 4616.454226] ? nd_jump_root+0xe5/0x160 [ 4616.457080] ? path_init+0x437/0x520 [ 4616.460699] path_openat+0xf7/0x500 [ 4616.462197] ? do_raw_spin_unlock+0x75/0x190 [ 4616.464342] do_filp_open+0x99/0x140 [ 4616.465771] ? getname_flags+0x6e/0x330 [ 4616.467664] ? __check_object_size+0xff/0x256 [ 4616.470441] ? do_raw_spin_unlock+0x75/0x190 [ 4616.472679] ? _raw_spin_unlock+0x12/0x30 [ 4616.475967] do_sys_openat2+0x2b4/0x410 [ 4616.479573] do_sys_open+0x73/0xa0 [ 4616.483622] __x64_sys_openat+0x24/0x30 [ 4616.488780] do_syscall_64+0xc1/0x440 [ 4616.490351] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4616.492911] RIP: 0033:0x7f1a63983332 [ 4616.496481] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4616.509296] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4616.516873] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4616.520102] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4616.525991] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4616.531111] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4616.535888] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4620.057071] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4620.065411] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4620.090330] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4620.096085] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4620.100151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4620.102820] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4620.120325] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4620.123746] RAX: 0000000000001dea RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4620.127855] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4620.131291] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4620.136173] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4620.138724] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4620.141029] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4620.144808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4620.148850] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4620.152373] Call Trace: [ 4620.153835] [ 4620.154940] ? show_regs.cold.9+0x22/0x2f [ 4620.156654] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4620.158778] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4620.160907] ? watchdog+0x40/0x40 [ 4620.162337] ? hrtimer_interrupt+0x128/0x2c0 [ 4620.166768] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4620.168552] ? apic_timer_interrupt+0xf/0x20 [ 4620.174389] [ 4620.176464] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4620.180100] do_raw_spin_lock+0x177/0x190 [ 4620.184272] _raw_spin_lock+0x21/0x30 [ 4620.186351] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4620.189387] ? do_raw_spin_unlock+0x75/0x190 [ 4620.191292] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4620.194556] ? do_wait_intr+0xf0/0xf0 [ 4620.198768] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4620.203810] mdc_close+0x53b/0xd70 [mdc] [ 4620.206848] lmv_close+0x18d/0x490 [lmv] [ 4620.210702] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4620.216352] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4620.220615] ll_md_close+0x274/0x800 [lustre] [ 4620.222680] ll_file_release+0x128/0xc50 [lustre] [ 4620.224987] __fput+0x10d/0x440 [ 4620.226846] ____fput+0x12/0x20 [ 4620.228310] task_work_run+0xc6/0x110 [ 4620.229915] exit_to_usermode_loop+0x1dd/0x1f0 [ 4620.231774] do_syscall_64+0x3d6/0x440 [ 4620.233384] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4620.235601] RIP: 0033:0x7f1a63983cd8 [ 4620.239155] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4620.251930] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4620.255863] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4620.259059] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4620.261445] BUG: workqueue lockup - pool [ 4620.262450] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4620.262460] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4620.262462] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4620.303830] cpus=0-3 flags=0x4 nice=0 stuck for 2597s! [ 4620.306180] Showing busy workqueues and worker pools: [ 4620.308073] workqueue events: flags=0x0 [ 4620.313235] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4620.317929] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4620.321711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4620.324213] pending: sec_gc_main [ptlrpc] [ 4620.328429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4620.333072] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4620.338812] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4620.344716] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4620.350754] workqueue events_long: flags=0x0 [ 4620.354082] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4620.360970] pending: lu_objects_flush [obdclass] [ 4620.364399] workqueue events_unbound: flags=0x2 [ 4620.366479] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4620.369276] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4620.377605] workqueue events_power_efficient: flags=0x80 [ 4620.379422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4620.381525] pending: neigh_periodic_work, do_cache_clean [ 4620.386833] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4620.390925] pending: neigh_periodic_work, check_lifetime [ 4620.394622] workqueue mm_percpu_wq: flags=0x8 [ 4620.396946] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4620.400686] pending: vmstat_update [ 4620.402580] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4620.406314] pending: vmstat_update [ 4620.408087] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4620.412871] pending: vmstat_update [ 4620.414957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4620.420559] pending: vmstat_update [ 4620.422576] workqueue writeback: flags=0x4a [ 4620.424912] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4620.429921] pending: wb_workfn, wb_workfn [ 4620.431560] workqueue xprtiod: flags=0xa [ 4620.433497] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4620.438665] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4620.443399] workqueue ipv6_addrconf: flags=0x40008 [ 4620.445192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4620.450532] pending: addrconf_verify_work [ 4620.454484] workqueue knbd0-recv: flags=0x1a [ 4620.456302] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4620.459803] in-flight: 647:recv_work [ 4620.461138] workqueue ptlrpc_pinger: flags=0x2 [ 4620.463566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4620.466453] pending: ptlrpc_pinger_main [ptlrpc] [ 4620.469109] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4610s workers=2 idle: 652 [ 4628.056031] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4628.058010] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4628.062832] Modules linked in: [ 4628.066611] Modules linked in: [ 4628.066614] lustre(O) osp(O) [ 4628.068382] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4628.068454] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4628.068461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4628.068464] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4628.068479] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4628.068483] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4628.068487] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4628.068490] RDX: 0000000000001c7b RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4628.068492] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4628.068494] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4628.068497] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4628.068500] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4628.068503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4628.068505] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4628.068515] Call Trace: [ 4628.068520] [ 4628.068526] ? show_regs.cold.9+0x22/0x2f [ 4628.068537] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4628.068543] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4628.068547] ? watchdog+0x40/0x40 [ 4628.068552] ? hrtimer_interrupt+0x128/0x2c0 [ 4628.068556] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4628.068564] ? apic_timer_interrupt+0xf/0x20 [ 4628.068569] [ 4628.068572] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4628.068577] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4628.068582] do_raw_spin_lock+0x177/0x190 [ 4628.068587] _raw_spin_lock+0x21/0x30 [ 4628.068593] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4628.070428] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4628.070497] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4628.070504] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4628.070506] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4628.070516] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4628.070520] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4628.070525] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4628.070527] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4628.070530] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4628.070532] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4628.070535] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4628.070537] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4628.070540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4628.070542] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4628.070550] Call Trace: [ 4628.070555] [ 4628.070560] ? show_regs.cold.9+0x22/0x2f [ 4628.070568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4628.070574] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4628.070578] ? watchdog+0x40/0x40 [ 4628.070582] ? hrtimer_interrupt+0x128/0x2c0 [ 4628.070586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4628.070591] ? apic_timer_interrupt+0xf/0x20 [ 4628.070596] [ 4628.070598] ? native_safe_halt+0x2e/0x50 [ 4628.070603] kvm_wait+0xcd/0xe0 [ 4628.070610] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4628.070617] do_raw_spin_lock+0x177/0x190 [ 4628.070621] _raw_spin_lock+0x21/0x30 [ 4628.070626] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4628.070883] ? do_raw_spin_unlock+0x75/0x190 [ 4628.070889] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4628.075332] obd_mod_free+0x46/0x140 [ptlrpc] [ 4628.075623] mdc_commit_open+0x91/0xb0 [mdc] [ 4628.116332] ? do_wait_intr+0xf0/0xf0 [ 4628.116353] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4628.116643] mdc_close+0x53b/0xd70 [mdc] [ 4628.116682] lmv_close+0x18d/0x490 [lmv] [ 4628.116713] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4628.116827] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4628.116912] ll_md_close+0x274/0x800 [lustre] [ 4628.116997] ll_file_release+0x128/0xc50 [lustre] [ 4628.117092] __fput+0x10d/0x440 [ 4628.117100] ____fput+0x12/0x20 [ 4628.117104] task_work_run+0xc6/0x110 [ 4628.117111] exit_to_usermode_loop+0x1dd/0x1f0 [ 4628.117117] do_syscall_64+0x3d6/0x440 [ 4628.117121] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4628.117129] RIP: 0033:0x7f1a63983cd8 [ 4628.117135] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4628.117139] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4628.117144] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4628.117146] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4628.117148] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4628.117150] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4628.117153] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4628.532215] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4628.539137] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4628.545790] after_reply+0xe2c/0x1780 [ptlrpc] [ 4628.549521] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4628.553440] ? __next_timer_interrupt+0x160/0x160 [ 4628.559210] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4628.563856] ? do_wait_intr+0xf0/0xf0 [ 4628.571591] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4628.587317] kthread+0x1d1/0x200 [ 4628.588862] ? set_kthread_struct+0x70/0x70 [ 4628.602650] ret_from_fork+0x1f/0x30 [ 4648.053128] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4648.057113] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4648.058010] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [ 4648.061127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4648.061272] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4648.061279] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4648.061282] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4648.061298] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4648.061302] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4648.061306] RAX: 0000000000002944 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4648.061309] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4648.061311] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4648.061313] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4648.061315] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4648.061318] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4648.061321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4648.061323] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4648.061332] Call Trace: [ 4648.061337] [ 4648.061343] ? show_regs.cold.9+0x22/0x2f [ 4648.061353] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4648.061360] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4648.061365] ? watchdog+0x40/0x40 [ 4648.061369] ? hrtimer_interrupt+0x128/0x2c0 [ 4648.061372] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4648.061380] ? apic_timer_interrupt+0xf/0x20 [ 4648.061384] [ 4648.061386] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4648.061392] do_raw_spin_lock+0x177/0x190 [ 4648.061396] _raw_spin_lock+0x21/0x30 [ 4648.061401] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4648.065063] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw [ 4648.090176] ? do_raw_spin_unlock+0x75/0x190 [ 4648.096743] libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4648.096767] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4648.096984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4648.096989] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4648.097004] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4648.097008] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4648.097014] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4648.097016] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4648.097018] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4648.097020] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4648.097023] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4648.097026] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4648.097029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4648.097031] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4648.097039] Call Trace: [ 4648.097043] [ 4648.097049] ? show_regs.cold.9+0x22/0x2f [ 4648.097058] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4648.097064] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4648.097068] ? watchdog+0x40/0x40 [ 4648.097073] ? hrtimer_interrupt+0x128/0x2c0 [ 4648.097077] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4648.097083] ? apic_timer_interrupt+0xf/0x20 [ 4648.097087] [ 4648.097090] ? native_safe_halt+0x2e/0x50 [ 4648.097094] kvm_wait+0xcd/0xe0 [ 4648.097101] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4648.097110] do_raw_spin_lock+0x177/0x190 [ 4648.097115] _raw_spin_lock+0x21/0x30 [ 4648.097119] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4648.097617] ? do_raw_spin_unlock+0x75/0x190 [ 4648.097622] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4648.098063] ? do_wait_intr+0xf0/0xf0 [ 4648.098073] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4648.098384] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4648.098670] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4648.099180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4648.107711] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4648.108229] ? do_wait_intr+0xf0/0xf0 [ 4648.108239] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4648.108647] mdc_close+0x53b/0xd70 [mdc] [ 4648.109002] lmv_close+0x18d/0x490 [lmv] [ 4648.109035] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4648.109217] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4648.109516] ll_md_close+0x274/0x800 [lustre] [ 4648.109610] ll_file_release+0x128/0xc50 [lustre] [ 4648.109697] __fput+0x10d/0x440 [ 4648.109706] ____fput+0x12/0x20 [ 4648.109710] task_work_run+0xc6/0x110 [ 4648.109718] exit_to_usermode_loop+0x1dd/0x1f0 [ 4648.109726] do_syscall_64+0x3d6/0x440 [ 4648.109731] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4648.109743] RIP: 0033:0x7f1a63983cd8 [ 4648.109750] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4648.109754] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4648.109759] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4648.109762] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4648.109764] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4648.109766] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4648.109768] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4648.652368] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4648.657010] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4648.661235] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4648.667202] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4648.670820] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4648.675105] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4648.681636] lmv_intent_open+0x338/0x1150 [lmv] [ 4648.684656] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4648.689545] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4648.693809] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4648.700903] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4648.703882] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4648.714728] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4648.720610] ? do_raw_spin_unlock+0x75/0x190 [ 4648.727670] ? _raw_spin_unlock+0x12/0x30 [ 4648.730112] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4648.735594] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4648.740622] ll_atomic_open+0x36a/0xd70 [lustre] [ 4648.743595] lookup_open+0x356/0xb40 [ 4648.745915] do_last+0x313/0xfc0 [ 4648.749579] ? nd_jump_root+0xe5/0x160 [ 4648.753387] ? path_init+0x437/0x520 [ 4648.754807] path_openat+0xf7/0x500 [ 4648.756650] ? do_raw_spin_unlock+0x75/0x190 [ 4648.759727] do_filp_open+0x99/0x140 [ 4648.761117] ? getname_flags+0x6e/0x330 [ 4648.764488] ? __check_object_size+0xff/0x256 [ 4648.766739] ? do_raw_spin_unlock+0x75/0x190 [ 4648.770702] ? _raw_spin_unlock+0x12/0x30 [ 4648.774691] do_sys_openat2+0x2b4/0x410 [ 4648.784667] do_sys_open+0x73/0xa0 [ 4648.786647] __x64_sys_openat+0x24/0x30 [ 4648.799339] do_syscall_64+0xc1/0x440 [ 4648.805555] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4648.812203] RIP: 0033:0x7f1a63983332 [ 4648.814760] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4648.830123] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4648.839073] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4648.844483] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4648.848564] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4648.852383] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4648.855359] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4650.976030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2627s! [ 4650.979360] Showing busy workqueues and worker pools: [ 4650.986664] workqueue events: flags=0x0 [ 4650.991405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4650.995364] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4651.003941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4651.007091] pending: sec_gc_main [ptlrpc] [ 4651.011371] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4651.015128] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4651.022348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4651.026000] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4651.034889] workqueue events_long: flags=0x0 [ 4651.037948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4651.043890] pending: lu_objects_flush [obdclass] [ 4651.048065] workqueue events_unbound: flags=0x2 [ 4651.050414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4651.053476] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4651.060138] workqueue events_power_efficient: flags=0x80 [ 4651.064104] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4651.070649] pending: neigh_periodic_work, do_cache_clean [ 4651.076434] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4651.080372] pending: neigh_periodic_work, check_lifetime [ 4651.083384] workqueue mm_percpu_wq: flags=0x8 [ 4651.088943] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4651.091270] pending: vmstat_update [ 4651.093386] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4651.096461] pending: vmstat_update [ 4651.098563] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4651.106000] pending: vmstat_update [ 4651.109843] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4651.121028] pending: vmstat_update [ 4651.129143] workqueue writeback: flags=0x4a [ 4651.141262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4651.147252] pending: wb_workfn, wb_workfn [ 4651.155082] workqueue xprtiod: flags=0xa [ 4651.158221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4651.169531] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4651.177353] workqueue ipv6_addrconf: flags=0x40008 [ 4651.179930] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4651.183165] pending: addrconf_verify_work [ 4651.188451] workqueue knbd0-recv: flags=0x1a [ 4651.192180] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4651.195080] in-flight: 647:recv_work [ 4651.202080] workqueue ptlrpc_pinger: flags=0x2 [ 4651.203764] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4651.207090] pending: ptlrpc_pinger_main [ptlrpc] [ 4651.211350] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4641s workers=2 idle: 652 [ 4656.056026] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4656.058224] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4656.058826] Modules linked in: lustre(O) [ 4656.063870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4656.063954] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4656.063962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4656.063965] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4656.063979] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4656.063983] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4656.063988] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4656.063990] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4656.063991] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4656.063993] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4656.063995] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4656.063998] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4656.064001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4656.064003] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4656.064011] Call Trace: [ 4656.064017] [ 4656.064023] ? show_regs.cold.9+0x22/0x2f [ 4656.064034] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4656.064041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4656.064045] ? watchdog+0x40/0x40 [ 4656.064049] ? hrtimer_interrupt+0x128/0x2c0 [ 4656.064053] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4656.064061] ? apic_timer_interrupt+0xf/0x20 [ 4656.064066] [ 4656.064069] ? native_safe_halt+0x2e/0x50 [ 4656.064073] kvm_wait+0xcd/0xe0 [ 4656.064080] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4656.064088] do_raw_spin_lock+0x177/0x190 [ 4656.064093] _raw_spin_lock+0x21/0x30 [ 4656.064098] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4656.064384] obd_mod_free+0x46/0x140 [ptlrpc] [ 4656.064577] mdc_commit_open+0x91/0xb0 [mdc] [ 4656.064614] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4656.069640] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4656.069726] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4656.069733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4656.069736] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4656.069749] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4656.069753] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4656.069757] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4656.069759] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4656.069777] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4656.069779] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4656.069782] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4656.069785] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4656.069788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4656.069791] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4656.069799] Call Trace: [ 4656.069804] [ 4656.069810] ? show_regs.cold.9+0x22/0x2f [ 4656.069821] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4656.069828] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4656.069833] ? watchdog+0x40/0x40 [ 4656.069837] ? hrtimer_interrupt+0x128/0x2c0 [ 4656.069841] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4656.069848] ? apic_timer_interrupt+0xf/0x20 [ 4656.069852] [ 4656.069854] ? native_safe_halt+0x2e/0x50 [ 4656.069858] kvm_wait+0xcd/0xe0 [ 4656.069866] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4656.069873] do_raw_spin_lock+0x177/0x190 [ 4656.069877] _raw_spin_lock+0x21/0x30 [ 4656.069882] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4656.070160] ? do_raw_spin_unlock+0x75/0x190 [ 4656.070167] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4656.070379] ? do_wait_intr+0xf0/0xf0 [ 4656.070388] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4656.070602] mdc_close+0x53b/0xd70 [mdc] [ 4656.070649] lmv_close+0x18d/0x490 [lmv] [ 4656.070680] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4656.071685] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4656.071933] ll_md_close+0x274/0x800 [lustre] [ 4656.072114] ll_file_release+0x128/0xc50 [lustre] [ 4656.072331] __fput+0x10d/0x440 [ 4656.072341] ____fput+0x12/0x20 [ 4656.072345] task_work_run+0xc6/0x110 [ 4656.072352] exit_to_usermode_loop+0x1dd/0x1f0 [ 4656.072361] do_syscall_64+0x3d6/0x440 [ 4656.072367] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4656.072376] RIP: 0033:0x7f1a63983cd8 [ 4656.072384] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4656.072388] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4656.072393] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4656.072395] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4656.072397] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4656.072400] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4656.072402] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4656.680586] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4656.686941] after_reply+0xe2c/0x1780 [ptlrpc] [ 4656.691277] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4656.697960] ? __next_timer_interrupt+0x160/0x160 [ 4656.699370] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4656.701584] ? do_wait_intr+0xf0/0xf0 [ 4656.703195] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4656.705898] kthread+0x1d1/0x200 [ 4656.707209] ? set_kthread_struct+0x70/0x70 [ 4656.709038] ret_from_fork+0x1f/0x30 [ 4676.053068] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4676.062755] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4676.068287] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4676.137411] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4676.137425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4676.137428] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4676.142955] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4676.143040] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4676.143049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4676.143052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4676.143067] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4676.143071] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4676.143076] RAX: 0000000000004e10 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4676.143079] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4676.143081] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4676.143083] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4676.143086] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4676.143090] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4676.143093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4676.143095] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4676.143561] Call Trace: [ 4676.143634] [ 4676.143641] ? show_regs.cold.9+0x22/0x2f [ 4676.143651] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4676.143657] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4676.143662] ? watchdog+0x40/0x40 [ 4676.143666] ? hrtimer_interrupt+0x128/0x2c0 [ 4676.143670] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4676.143678] ? apic_timer_interrupt+0xf/0x20 [ 4676.143682] [ 4676.143685] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4676.143691] do_raw_spin_lock+0x177/0x190 [ 4676.143696] _raw_spin_lock+0x21/0x30 [ 4676.143700] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4676.144167] ? do_raw_spin_unlock+0x75/0x190 [ 4676.144173] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4676.144505] ? do_wait_intr+0xf0/0xf0 [ 4676.144512] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4676.144981] mdc_close+0x53b/0xd70 [mdc] [ 4676.145022] lmv_close+0x18d/0x490 [lmv] [ 4676.145052] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4676.145169] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4676.145321] ll_md_close+0x274/0x800 [lustre] [ 4676.145475] ll_file_release+0x128/0xc50 [lustre] [ 4676.145557] __fput+0x10d/0x440 [ 4676.145643] ____fput+0x12/0x20 [ 4676.145649] task_work_run+0xc6/0x110 [ 4676.145657] exit_to_usermode_loop+0x1dd/0x1f0 [ 4676.145664] do_syscall_64+0x3d6/0x440 [ 4676.145669] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4676.145676] RIP: 0033:0x7f1a63983cd8 [ 4676.145685] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4676.145689] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4676.145694] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4676.145696] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4676.145698] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4676.145700] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4676.145702] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4676.638696] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4676.660750] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4676.666678] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4676.670063] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4676.673901] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4676.679507] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4676.684107] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4676.688542] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4676.695244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4676.703780] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4676.711999] Call Trace: [ 4676.716059] [ 4676.718230] ? show_regs.cold.9+0x22/0x2f [ 4676.720553] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4676.724749] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4676.731437] ? watchdog+0x40/0x40 [ 4676.734677] ? hrtimer_interrupt+0x128/0x2c0 [ 4676.740031] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4676.751847] ? apic_timer_interrupt+0xf/0x20 [ 4676.757774] [ 4676.759713] ? native_safe_halt+0x2e/0x50 [ 4676.766030] kvm_wait+0xcd/0xe0 [ 4676.769522] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4676.778303] do_raw_spin_lock+0x177/0x190 [ 4676.784455] _raw_spin_lock+0x21/0x30 [ 4676.786908] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4676.795356] ? do_raw_spin_unlock+0x75/0x190 [ 4676.800955] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4676.805645] ? do_wait_intr+0xf0/0xf0 [ 4676.807493] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4676.815715] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4676.822375] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4676.826861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4676.837753] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4676.849064] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4676.856802] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4676.862275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4676.869130] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4676.871524] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4676.877310] lmv_intent_open+0x338/0x1150 [lmv] [ 4676.879762] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4676.889146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4676.895862] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4676.901825] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4676.906928] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4676.911521] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4676.916287] ? do_raw_spin_unlock+0x75/0x190 [ 4676.921500] ? _raw_spin_unlock+0x12/0x30 [ 4676.926406] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4676.932881] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4676.938692] ll_atomic_open+0x36a/0xd70 [lustre] [ 4676.942897] lookup_open+0x356/0xb40 [ 4676.946998] do_last+0x313/0xfc0 [ 4676.948204] ? nd_jump_root+0xe5/0x160 [ 4676.952515] ? path_init+0x437/0x520 [ 4676.955774] path_openat+0xf7/0x500 [ 4676.962691] ? do_raw_spin_unlock+0x75/0x190 [ 4676.968083] do_filp_open+0x99/0x140 [ 4676.975470] ? getname_flags+0x6e/0x330 [ 4676.980899] ? __check_object_size+0xff/0x256 [ 4676.983153] ? do_raw_spin_unlock+0x75/0x190 [ 4676.985912] ? _raw_spin_unlock+0x12/0x30 [ 4676.987911] do_sys_openat2+0x2b4/0x410 [ 4676.990768] do_sys_open+0x73/0xa0 [ 4676.992806] __x64_sys_openat+0x24/0x30 [ 4676.994870] do_syscall_64+0xc1/0x440 [ 4676.996746] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4676.999071] RIP: 0033:0x7f1a63983332 [ 4677.000896] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4677.009010] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4677.011652] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4677.014402] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4677.017349] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4677.020575] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4677.023883] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4681.708627] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2658s! [ 4681.717474] Showing busy workqueues and worker pools: [ 4681.720882] workqueue events: flags=0x0 [ 4681.726969] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4681.742676] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4681.753563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4681.760820] pending: sec_gc_main [ptlrpc] [ 4681.767001] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4681.784205] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4681.809848] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4681.822897] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4681.829231] workqueue events_long: flags=0x0 [ 4681.833560] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4681.838031] pending: lu_objects_flush [obdclass] [ 4681.846175] workqueue events_unbound: flags=0x2 [ 4681.854161] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4681.865605] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4681.890478] workqueue events_power_efficient: flags=0x80 [ 4681.899503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4681.904102] pending: neigh_periodic_work, do_cache_clean [ 4681.911012] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4681.923020] pending: neigh_periodic_work, check_lifetime [ 4681.929425] workqueue mm_percpu_wq: flags=0x8 [ 4681.935970] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4681.947426] pending: vmstat_update [ 4681.952779] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4681.964148] pending: vmstat_update [ 4681.966969] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4681.975972] pending: vmstat_update [ 4681.980800] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4681.994797] pending: vmstat_update [ 4682.001850] workqueue writeback: flags=0x4a [ 4682.011157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4682.024966] pending: wb_workfn, wb_workfn [ 4682.041067] workqueue xprtiod: flags=0xa [ 4682.042723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4682.051472] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4682.056946] workqueue ipv6_addrconf: flags=0x40008 [ 4682.072495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4682.078953] pending: addrconf_verify_work [ 4682.080615] workqueue knbd0-recv: flags=0x1a [ 4682.092031] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4682.106166] in-flight: 647:recv_work [ 4682.109898] workqueue ptlrpc_pinger: flags=0x2 [ 4682.116195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4682.123391] pending: ptlrpc_pinger_main [ptlrpc] [ 4682.132701] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4672s workers=2 idle: 652 [ 4684.056785] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4684.060258] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4684.061595] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4684.092730] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4684.096094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4684.096172] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4684.096179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4684.096182] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4684.096196] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4684.096201] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4684.096205] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4684.096208] RDX: 0000000000000c35 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4684.096210] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4684.096212] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4684.096215] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4684.096218] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4684.096221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4684.096224] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4684.096232] Call Trace: [ 4684.096267] [ 4684.096272] ? show_regs.cold.9+0x22/0x2f [ 4684.096282] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4684.096289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4684.096296] ? watchdog+0x40/0x40 [ 4684.096301] ? hrtimer_interrupt+0x128/0x2c0 [ 4684.096306] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4684.096317] ? apic_timer_interrupt+0xf/0x20 [ 4684.096322] [ 4684.096325] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4684.096329] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4684.096334] do_raw_spin_lock+0x177/0x190 [ 4684.096339] _raw_spin_lock+0x21/0x30 [ 4684.096344] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4684.096659] obd_mod_free+0x46/0x140 [ptlrpc] [ 4684.110050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4684.131215] mdc_commit_open+0x91/0xb0 [mdc] [ 4684.136484] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4684.136523] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4684.136527] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4684.136533] RAX: 0000000000004a06 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4684.136535] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4684.136538] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4684.136539] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4684.136542] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4684.136545] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4684.136548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4684.136550] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4684.136560] Call Trace: [ 4684.136565] [ 4684.136570] ? show_regs.cold.9+0x22/0x2f [ 4684.136578] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4684.136585] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4684.136590] ? watchdog+0x40/0x40 [ 4684.136594] ? hrtimer_interrupt+0x128/0x2c0 [ 4684.136598] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4684.136607] ? apic_timer_interrupt+0xf/0x20 [ 4684.136612] [ 4684.136615] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4684.136622] do_raw_spin_lock+0x177/0x190 [ 4684.136625] _raw_spin_lock+0x21/0x30 [ 4684.136630] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4684.137067] ? do_raw_spin_unlock+0x75/0x190 [ 4684.137073] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4684.137276] ? do_wait_intr+0xf0/0xf0 [ 4684.137285] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4684.137482] mdc_close+0x53b/0xd70 [mdc] [ 4684.137652] lmv_close+0x18d/0x490 [lmv] [ 4684.137692] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4684.137862] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4684.138077] ll_md_close+0x274/0x800 [lustre] [ 4684.138175] ll_file_release+0x128/0xc50 [lustre] [ 4684.138265] __fput+0x10d/0x440 [ 4684.138274] ____fput+0x12/0x20 [ 4684.138278] task_work_run+0xc6/0x110 [ 4684.138287] exit_to_usermode_loop+0x1dd/0x1f0 [ 4684.138294] do_syscall_64+0x3d6/0x440 [ 4684.138298] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4684.138305] RIP: 0033:0x7f1a63983cd8 [ 4684.138325] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4684.138330] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4684.138334] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4684.138337] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4684.138340] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4684.138342] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4684.138344] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4684.435223] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4684.437832] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4684.440979] after_reply+0xe2c/0x1780 [ptlrpc] [ 4684.443850] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4684.446374] ? __next_timer_interrupt+0x160/0x160 [ 4684.448964] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4684.451445] ? do_wait_intr+0xf0/0xf0 [ 4684.453452] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4684.456408] kthread+0x1d1/0x200 [ 4684.458176] ? set_kthread_struct+0x70/0x70 [ 4684.460359] ret_from_fork+0x1f/0x30 [ 4704.067609] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 4704.073771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4704.104953] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4704.111792] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4704.116962] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4704.121378] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4704.132133] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4704.137167] RAX: 0000000000003208 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4704.141872] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4704.146396] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4704.150690] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4704.155612] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4704.161004] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4704.166210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4704.169526] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4704.174181] Call Trace: [ 4704.175966] [ 4704.177299] ? show_regs.cold.9+0x22/0x2f [ 4704.180112] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4704.183292] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4704.186513] ? watchdog+0x40/0x40 [ 4704.188800] ? hrtimer_interrupt+0x128/0x2c0 [ 4704.191744] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4704.194616] ? apic_timer_interrupt+0xf/0x20 [ 4704.196755] [ 4704.197934] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4704.201426] do_raw_spin_lock+0x177/0x190 [ 4704.205170] _raw_spin_lock+0x21/0x30 [ 4704.207841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4704.211198] ? do_raw_spin_unlock+0x75/0x190 [ 4704.214098] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4704.217099] ? do_wait_intr+0xf0/0xf0 [ 4704.219460] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4704.223641] mdc_close+0x53b/0xd70 [mdc] [ 4704.226124] lmv_close+0x18d/0x490 [lmv] [ 4704.228488] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4704.232677] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4704.236451] ll_md_close+0x274/0x800 [lustre] [ 4704.239440] ll_file_release+0x128/0xc50 [lustre] [ 4704.242311] __fput+0x10d/0x440 [ 4704.247572] ____fput+0x12/0x20 [ 4704.249092] task_work_run+0xc6/0x110 [ 4704.251382] exit_to_usermode_loop+0x1dd/0x1f0 [ 4704.253725] do_syscall_64+0x3d6/0x440 [ 4704.257659] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4704.260679] RIP: 0033:0x7f1a63983cd8 [ 4704.265273] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4704.286882] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4704.299481] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4704.306923] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4704.311467] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4704.319601] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4704.325549] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4708.065908] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4708.077020] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4708.150490] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4708.166495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4708.184025] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4708.192204] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4708.222754] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4708.237575] RAX: 0000000000006685 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4708.249780] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4708.263741] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4708.269631] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4708.280434] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4708.288045] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4708.309587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4708.324185] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4708.347918] Call Trace: [ 4708.356753] [ 4708.371381] ? show_regs.cold.9+0x22/0x2f [ 4708.385777] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4708.394859] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4708.410194] ? watchdog+0x40/0x40 [ 4708.417957] ? hrtimer_interrupt+0x128/0x2c0 [ 4708.420459] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4708.433794] ? apic_timer_interrupt+0xf/0x20 [ 4708.445171] [ 4708.450079] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4708.455525] do_raw_spin_lock+0x177/0x190 [ 4708.458049] _raw_spin_lock+0x21/0x30 [ 4708.466277] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4708.471412] ? do_raw_spin_unlock+0x75/0x190 [ 4708.480002] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4708.490069] ? do_wait_intr+0xf0/0xf0 [ 4708.494645] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4708.507787] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4708.515065] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4708.538100] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4708.547618] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4708.556974] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4708.558799] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4708.565973] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4708.572519] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4708.577390] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4708.581995] lmv_intent_open+0x338/0x1150 [lmv] [ 4708.587295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4708.588929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4708.610933] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4708.618660] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4708.640823] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4708.651582] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4708.666106] ? do_raw_spin_unlock+0x75/0x190 [ 4708.679176] ? _raw_spin_unlock+0x12/0x30 [ 4708.680776] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4708.692027] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4708.703092] ll_atomic_open+0x36a/0xd70 [lustre] [ 4708.709594] lookup_open+0x356/0xb40 [ 4708.714248] do_last+0x313/0xfc0 [ 4708.716581] ? nd_jump_root+0xe5/0x160 [ 4708.720269] ? path_init+0x437/0x520 [ 4708.727602] path_openat+0xf7/0x500 [ 4708.732096] ? do_raw_spin_unlock+0x75/0x190 [ 4708.742849] do_filp_open+0x99/0x140 [ 4708.744087] ? getname_flags+0x6e/0x330 [ 4708.751557] ? __check_object_size+0xff/0x256 [ 4708.754972] ? do_raw_spin_unlock+0x75/0x190 [ 4708.759043] ? _raw_spin_unlock+0x12/0x30 [ 4708.765159] do_sys_openat2+0x2b4/0x410 [ 4708.766588] do_sys_open+0x73/0xa0 [ 4708.771880] __x64_sys_openat+0x24/0x30 [ 4708.774524] do_syscall_64+0xc1/0x440 [ 4708.787582] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4708.805986] RIP: 0033:0x7f1a63983332 [ 4708.811824] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4708.837069] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4708.858147] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4708.871858] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4708.902541] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4708.928920] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4708.947118] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4712.056878] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4712.060997] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4712.071238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4712.071325] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4712.071332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4712.071336] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4712.071353] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4712.071357] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4712.071361] RAX: 000000000000138f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4712.071364] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4712.071365] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4712.071367] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4712.071370] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4712.071373] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4712.071375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4712.071377] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4712.071386] Call Trace: [ 4712.071392] [ 4712.071398] ? show_regs.cold.9+0x22/0x2f [ 4712.071408] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4712.071415] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4712.071419] ? watchdog+0x40/0x40 [ 4712.071423] ? hrtimer_interrupt+0x128/0x2c0 [ 4712.071428] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4712.071437] ? apic_timer_interrupt+0xf/0x20 [ 4712.071441] [ 4712.071444] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4712.071450] do_raw_spin_lock+0x177/0x190 [ 4712.071455] _raw_spin_lock+0x21/0x30 [ 4712.071460] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4712.071719] ? do_raw_spin_unlock+0x75/0x190 [ 4712.071725] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4712.071923] ? do_wait_intr+0xf0/0xf0 [ 4712.071931] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4712.072126] mdc_close+0x53b/0xd70 [mdc] [ 4712.072163] lmv_close+0x18d/0x490 [lmv] [ 4712.072197] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4712.072334] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4712.072416] ll_md_close+0x274/0x800 [lustre] [ 4712.072498] ll_file_release+0x128/0xc50 [lustre] [ 4712.072580] __fput+0x10d/0x440 [ 4712.072591] ____fput+0x12/0x20 [ 4712.072595] task_work_run+0xc6/0x110 [ 4712.072602] exit_to_usermode_loop+0x1dd/0x1f0 [ 4712.072609] do_syscall_64+0x3d6/0x440 [ 4712.072613] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4712.072621] RIP: 0033:0x7f1a63983cd8 [ 4712.072628] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4712.072632] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4712.072636] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4712.072639] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4712.072641] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4712.072643] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4712.072645] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4712.418553] BUG: workqueue lockup - pool [ 4712.419465] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4712.419712] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4712.419722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4712.419725] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4712.419742] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4712.419746] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4712.419751] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4712.419753] RDX: 00000000000069d5 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4712.419755] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4712.419758] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4712.419760] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4712.419763] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4712.419766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4712.419768] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4712.419778] Call Trace: [ 4712.419784] [ 4712.419790] ? show_regs.cold.9+0x22/0x2f [ 4712.419801] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4712.419809] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4712.419813] ? watchdog+0x40/0x40 [ 4712.419818] ? hrtimer_interrupt+0x128/0x2c0 [ 4712.419822] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4712.419829] ? apic_timer_interrupt+0xf/0x20 [ 4712.419833] [ 4712.419836] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4712.419841] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4712.419845] do_raw_spin_lock+0x177/0x190 [ 4712.419850] _raw_spin_lock+0x21/0x30 [ 4712.419855] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4712.422848] cpus=0-3 flags=0x4 nice=0 stuck for 2689s! [ 4712.422868] Showing busy workqueues and worker pools: [ 4712.422871] workqueue events: flags=0x0 [ 4712.422874] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4712.422883] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4712.423336] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4712.423348] pending: sec_gc_main [ptlrpc] [ 4712.423750] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4712.423759] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4712.425763] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4712.425777] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4712.432874] obd_mod_free+0x46/0x140 [ptlrpc] [ 4712.433180] mdc_commit_open+0x91/0xb0 [mdc] [ 4712.433231] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4712.433448] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4712.433656] after_reply+0xe2c/0x1780 [ptlrpc] [ 4712.433861] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4712.448037] [ 4712.448076] workqueue events_long: flags=0x0 [ 4712.448084] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4712.448182] pending: lu_objects_flush [obdclass] [ 4712.448453] workqueue events_unbound: flags=0x2 [ 4712.448456] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4712.448463] pending: flush_memcg_stats_dwork [ 4712.463452] ? __next_timer_interrupt+0x160/0x160 [ 4712.467942] , idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4712.468013] workqueue events_power_efficient: flags=0x80 [ 4712.468019] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4712.468032] pending: neigh_periodic_work, do_cache_clean [ 4712.468048] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4712.468056] pending: neigh_periodic_work, check_lifetime [ 4712.468069] workqueue mm_percpu_wq: flags=0x8 [ 4712.468072] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4712.468077] pending: vmstat_update [ 4712.468084] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4712.468091] pending: vmstat_update [ 4712.468095] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4712.468102] pending: vmstat_update [ 4712.468106] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4712.468113] pending: vmstat_update [ 4712.468121] workqueue writeback: flags=0x4a [ 4712.468123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4712.468129] pending: wb_workfn, wb_workfn [ 4712.468147] workqueue xprtiod: flags=0xa [ 4712.468149] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4712.468155] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4712.468176] workqueue ipv6_addrconf: flags=0x40008 [ 4712.468178] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4712.468184] pending: addrconf_verify_work [ 4712.468197] workqueue knbd0-recv: flags=0x1a [ 4712.468200] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4712.468207] in-flight: 647:recv_work [ 4712.468232] workqueue ptlrpc_pinger: flags=0x2 [ 4712.468235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4712.468240] pending: ptlrpc_pinger_main [ptlrpc] [ 4712.468543] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4702s workers=2 idle: 652 [ 4712.741849] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4712.743830] ? do_wait_intr+0xf0/0xf0 [ 4712.745403] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4712.747621] kthread+0x1d1/0x200 [ 4712.748652] ? set_kthread_struct+0x70/0x70 [ 4712.750085] ret_from_fork+0x1f/0x30 [ 4732.057214] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 4732.070668] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4732.108763] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4732.121196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4732.129805] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4732.133709] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4732.154962] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4732.160710] RAX: 0000000000002089 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4732.168916] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4732.174007] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4732.180588] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4732.183817] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4732.188015] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4732.197635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4732.204207] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4732.210591] Call Trace: [ 4732.211670] [ 4732.215565] ? show_regs.cold.9+0x22/0x2f [ 4732.216815] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4732.226617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4732.229794] ? watchdog+0x40/0x40 [ 4732.232592] ? hrtimer_interrupt+0x128/0x2c0 [ 4732.236308] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4732.238043] ? apic_timer_interrupt+0xf/0x20 [ 4732.245880] [ 4732.247218] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4732.249870] do_raw_spin_lock+0x177/0x190 [ 4732.252222] _raw_spin_lock+0x21/0x30 [ 4732.258253] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4732.261839] ? do_raw_spin_unlock+0x75/0x190 [ 4732.267645] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4732.278002] ? do_wait_intr+0xf0/0xf0 [ 4732.279466] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4732.293762] mdc_close+0x53b/0xd70 [mdc] [ 4732.297592] lmv_close+0x18d/0x490 [lmv] [ 4732.300911] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4732.303163] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4732.310753] ll_md_close+0x274/0x800 [lustre] [ 4732.314602] ll_file_release+0x128/0xc50 [lustre] [ 4732.318990] __fput+0x10d/0x440 [ 4732.321380] ____fput+0x12/0x20 [ 4732.323282] task_work_run+0xc6/0x110 [ 4732.328272] exit_to_usermode_loop+0x1dd/0x1f0 [ 4732.330796] do_syscall_64+0x3d6/0x440 [ 4732.334540] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4732.338955] RIP: 0033:0x7f1a63983cd8 [ 4732.342704] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4732.360782] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4732.365177] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4732.372616] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4732.379592] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4732.387677] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4732.393226] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4740.053042] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4740.056029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4740.056044] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4740.056134] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4740.056142] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4740.056145] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4740.056159] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4740.056165] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4740.056169] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4740.056171] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4740.056173] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4740.056175] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4740.056178] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4740.056181] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4740.056192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4740.056194] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4740.056202] Call Trace: [ 4740.056208] [ 4740.056214] ? show_regs.cold.9+0x22/0x2f [ 4740.056226] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4740.056233] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4740.056238] ? watchdog+0x40/0x40 [ 4740.056242] ? hrtimer_interrupt+0x128/0x2c0 [ 4740.056253] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4740.056260] ? apic_timer_interrupt+0xf/0x20 [ 4740.056264] [ 4740.056268] ? native_safe_halt+0x2e/0x50 [ 4740.056271] kvm_wait+0xcd/0xe0 [ 4740.056282] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4740.056293] do_raw_spin_lock+0x177/0x190 [ 4740.056298] _raw_spin_lock+0x21/0x30 [ 4740.056308] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4740.058322] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4740.058331] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4740.058397] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4740.058402] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4740.058405] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4740.058451] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4740.058456] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4740.058464] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4740.058467] RDX: 0000000000007bb4 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4740.058469] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4740.058471] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4740.058473] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4740.058477] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4740.058479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4740.058481] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4740.058488] Call Trace: [ 4740.058492] [ 4740.058496] ? show_regs.cold.9+0x22/0x2f [ 4740.058505] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4740.058511] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4740.058515] ? watchdog+0x40/0x40 [ 4740.058520] ? hrtimer_interrupt+0x128/0x2c0 [ 4740.058529] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4740.058536] ? apic_timer_interrupt+0xf/0x20 [ 4740.058541] [ 4740.058543] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4740.058548] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4740.058553] do_raw_spin_lock+0x177/0x190 [ 4740.058557] _raw_spin_lock+0x21/0x30 [ 4740.058562] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4740.058829] obd_mod_free+0x46/0x140 [ptlrpc] [ 4740.059059] mdc_commit_open+0x91/0xb0 [mdc] [ 4740.059096] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4740.059287] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4740.059478] after_reply+0xe2c/0x1780 [ptlrpc] [ 4740.059668] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4740.059861] ? __next_timer_interrupt+0x160/0x160 [ 4740.059870] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4740.060082] ? do_wait_intr+0xf0/0xf0 [ 4740.060090] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4740.060287] kthread+0x1d1/0x200 [ 4740.060292] ? set_kthread_struct+0x70/0x70 [ 4740.060295] ret_from_fork+0x1f/0x30 [ 4740.062296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4740.062371] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4740.062378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4740.062381] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4740.062392] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4740.062396] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4740.062401] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4740.062403] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4740.062405] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4740.062407] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4740.062409] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4740.062411] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4740.062414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4740.062416] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4740.062425] Call Trace: [ 4740.062430] [ 4740.062434] ? show_regs.cold.9+0x22/0x2f [ 4740.062444] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4740.062449] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4740.062452] ? watchdog+0x40/0x40 [ 4740.062456] ? hrtimer_interrupt+0x128/0x2c0 [ 4740.062460] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4740.062466] ? apic_timer_interrupt+0xf/0x20 [ 4740.062470] [ 4740.062473] ? native_safe_halt+0x2e/0x50 [ 4740.062477] kvm_wait+0xcd/0xe0 [ 4740.062483] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4740.062489] do_raw_spin_lock+0x177/0x190 [ 4740.062494] _raw_spin_lock+0x21/0x30 [ 4740.062497] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4740.081473] ? do_raw_spin_unlock+0x75/0x190 [ 4740.081493] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4740.081770] ? do_wait_intr+0xf0/0xf0 [ 4740.081779] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4740.081987] mdc_close+0x53b/0xd70 [mdc] [ 4740.082025] lmv_close+0x18d/0x490 [lmv] [ 4740.082054] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4740.082168] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4740.082248] ll_md_close+0x274/0x800 [lustre] [ 4740.082329] ll_file_release+0x128/0xc50 [lustre] [ 4740.082408] __fput+0x10d/0x440 [ 4740.082416] ____fput+0x12/0x20 [ 4740.082420] task_work_run+0xc6/0x110 [ 4740.082427] exit_to_usermode_loop+0x1dd/0x1f0 [ 4740.082433] do_syscall_64+0x3d6/0x440 [ 4740.082437] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4740.082444] RIP: 0033:0x7f1a63983cd8 [ 4740.082452] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4740.082456] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4740.082461] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4740.082463] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4740.082466] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4740.082468] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4740.082470] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4740.808870] ? do_raw_spin_unlock+0x75/0x190 [ 4740.813869] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4740.819069] ? do_wait_intr+0xf0/0xf0 [ 4740.820119] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4740.828087] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4740.832621] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4740.834748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4740.837516] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4740.840556] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4740.844534] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4740.846746] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4740.854822] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4740.857704] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4740.860471] lmv_intent_open+0x338/0x1150 [lmv] [ 4740.862606] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4740.875552] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4740.884654] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4740.889670] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4740.895449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4740.906128] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4740.907881] ? do_raw_spin_unlock+0x75/0x190 [ 4740.918444] ? _raw_spin_unlock+0x12/0x30 [ 4740.926597] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4740.930453] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4740.938040] ll_atomic_open+0x36a/0xd70 [lustre] [ 4740.939631] lookup_open+0x356/0xb40 [ 4740.944670] do_last+0x313/0xfc0 [ 4740.948385] ? nd_jump_root+0xe5/0x160 [ 4740.957293] ? path_init+0x437/0x520 [ 4740.961916] path_openat+0xf7/0x500 [ 4740.969357] ? do_raw_spin_unlock+0x75/0x190 [ 4740.975561] do_filp_open+0x99/0x140 [ 4740.981807] ? getname_flags+0x6e/0x330 [ 4740.983258] ? __check_object_size+0xff/0x256 [ 4740.987814] ? do_raw_spin_unlock+0x75/0x190 [ 4740.990192] ? _raw_spin_unlock+0x12/0x30 [ 4740.993497] do_sys_openat2+0x2b4/0x410 [ 4740.995149] do_sys_open+0x73/0xa0 [ 4740.997106] __x64_sys_openat+0x24/0x30 [ 4740.999388] do_syscall_64+0xc1/0x440 [ 4741.002706] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4741.005437] RIP: 0033:0x7f1a63983332 [ 4741.013382] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4741.036522] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4741.057316] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4741.060931] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4741.063410] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4741.067307] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4741.070703] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4742.625848] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2719s! [ 4742.629716] Showing busy workqueues and worker pools: [ 4742.631748] workqueue events: flags=0x0 [ 4742.633498] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4742.636986] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4742.640969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4742.645442] pending: sec_gc_main [ptlrpc] [ 4742.648666] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4742.651412] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4742.655837] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4742.659742] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4742.665136] workqueue events_long: flags=0x0 [ 4742.666807] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4742.670248] pending: lu_objects_flush [obdclass] [ 4742.672952] workqueue events_unbound: flags=0x2 [ 4742.674854] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4742.677892] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4742.682874] workqueue events_power_efficient: flags=0x80 [ 4742.685128] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4742.689002] pending: neigh_periodic_work, do_cache_clean [ 4742.691912] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4742.696071] pending: neigh_periodic_work, check_lifetime [ 4742.698625] workqueue mm_percpu_wq: flags=0x8 [ 4742.700908] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4742.703788] pending: vmstat_update [ 4742.705155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4742.708522] pending: vmstat_update [ 4742.710609] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4742.713003] pending: vmstat_update [ 4742.714941] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4742.717437] pending: vmstat_update [ 4742.719017] workqueue writeback: flags=0x4a [ 4742.721012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4742.723511] pending: wb_workfn, wb_workfn [ 4742.725554] workqueue xprtiod: flags=0xa [ 4742.727112] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4742.729753] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4742.732653] workqueue ipv6_addrconf: flags=0x40008 [ 4742.734487] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4742.737502] pending: addrconf_verify_work [ 4742.739449] workqueue knbd0-recv: flags=0x1a [ 4742.741620] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4742.744579] in-flight: 647:recv_work [ 4742.746680] workqueue ptlrpc_pinger: flags=0x2 [ 4742.748578] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4742.751274] pending: ptlrpc_pinger_main [ptlrpc] [ 4742.754331] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4732s workers=2 idle: 652 [ 4760.057307] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 4760.070860] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4760.091139] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4760.095360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4760.098887] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4760.101356] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4760.108969] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4760.111452] RAX: 000000000000580d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4760.114365] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4760.116815] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4760.119271] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4760.122075] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4760.125323] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4760.128482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4760.130861] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4760.133725] Call Trace: [ 4760.134871] [ 4760.135704] ? show_regs.cold.9+0x22/0x2f [ 4760.137371] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4760.139293] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4760.141168] ? watchdog+0x40/0x40 [ 4760.142498] ? hrtimer_interrupt+0x128/0x2c0 [ 4760.144191] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4760.145703] ? apic_timer_interrupt+0xf/0x20 [ 4760.147026] [ 4760.147495] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4760.149085] do_raw_spin_lock+0x177/0x190 [ 4760.150115] _raw_spin_lock+0x21/0x30 [ 4760.151011] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4760.152459] ? do_raw_spin_unlock+0x75/0x190 [ 4760.153505] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4760.155014] ? do_wait_intr+0xf0/0xf0 [ 4760.155999] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4760.157448] mdc_close+0x53b/0xd70 [mdc] [ 4760.158507] lmv_close+0x18d/0x490 [lmv] [ 4760.159575] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4760.161187] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4760.162402] ll_md_close+0x274/0x800 [lustre] [ 4760.163594] ll_file_release+0x128/0xc50 [lustre] [ 4760.164939] __fput+0x10d/0x440 [ 4760.165636] ____fput+0x12/0x20 [ 4760.166517] task_work_run+0xc6/0x110 [ 4760.167469] exit_to_usermode_loop+0x1dd/0x1f0 [ 4760.168777] do_syscall_64+0x3d6/0x440 [ 4760.169746] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4760.171268] RIP: 0033:0x7f1a63983cd8 [ 4760.172239] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4760.180863] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4760.182871] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4760.185255] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4760.188598] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4760.191669] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4760.199195] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4768.053262] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4768.056030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4768.058035] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4768.058046] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4768.058123] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4768.058129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4768.058132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4768.058147] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4768.058151] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4768.058156] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4768.058159] RDX: 0000000000002049 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4768.058161] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4768.058164] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4768.058166] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4768.058169] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4768.058172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4768.058175] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4768.058184] Call Trace: [ 4768.058190] [ 4768.058196] ? show_regs.cold.9+0x22/0x2f [ 4768.058206] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4768.058212] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4768.058217] ? watchdog+0x40/0x40 [ 4768.058222] ? hrtimer_interrupt+0x128/0x2c0 [ 4768.058226] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4768.058236] ? apic_timer_interrupt+0xf/0x20 [ 4768.058241] [ 4768.058244] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4768.058250] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4768.058255] do_raw_spin_lock+0x177/0x190 [ 4768.058261] _raw_spin_lock+0x21/0x30 [ 4768.058267] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4768.058635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4768.058851] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4768.058856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4768.058859] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4768.058868] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4768.058871] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4768.058876] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4768.058878] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4768.058880] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4768.058882] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4768.058883] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4768.058886] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4768.058889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4768.058891] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4768.058898] Call Trace: [ 4768.058901] [ 4768.058904] ? show_regs.cold.9+0x22/0x2f [ 4768.058912] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4768.058918] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4768.058921] ? watchdog+0x40/0x40 [ 4768.058926] ? hrtimer_interrupt+0x128/0x2c0 [ 4768.058930] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4768.058936] ? apic_timer_interrupt+0xf/0x20 [ 4768.058940] [ 4768.058943] ? native_safe_halt+0x2e/0x50 [ 4768.058947] kvm_wait+0xcd/0xe0 [ 4768.058956] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4768.058962] do_raw_spin_lock+0x177/0x190 [ 4768.058967] _raw_spin_lock+0x21/0x30 [ 4768.059115] obd_mod_free+0x46/0x140 [ptlrpc] [ 4768.058971] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4768.059636] ? do_raw_spin_unlock+0x75/0x190 [ 4768.059643] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4768.060097] mdc_commit_open+0x91/0xb0 [mdc] [ 4768.060192] ? do_wait_intr+0xf0/0xf0 [ 4768.060135] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4768.060201] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4768.060637] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4768.061098] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4768.061732] after_reply+0xe2c/0x1780 [ptlrpc] [ 4768.061930] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4768.062129] ? __next_timer_interrupt+0x160/0x160 [ 4768.062141] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4768.062343] ? do_wait_intr+0xf0/0xf0 [ 4768.062352] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4768.062550] kthread+0x1d1/0x200 [ 4768.062556] ? set_kthread_struct+0x70/0x70 [ 4768.062560] ret_from_fork+0x1f/0x30 [ 4768.061454] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4768.065483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4768.065590] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4768.065626] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4768.065657] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4768.065795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4768.065879] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4768.066175] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4768.066240] lmv_intent_open+0x338/0x1150 [lmv] [ 4768.066263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4768.066374] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4768.066429] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4768.066444] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4768.066515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4768.066599] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4768.066774] ? do_raw_spin_unlock+0x75/0x190 [ 4768.066781] ? _raw_spin_unlock+0x12/0x30 [ 4768.066787] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4768.066816] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4768.066836] ll_atomic_open+0x36a/0xd70 [lustre] [ 4768.066895] lookup_open+0x356/0xb40 [ 4768.066901] do_last+0x313/0xfc0 [ 4768.066905] ? nd_jump_root+0xe5/0x160 [ 4768.066908] ? path_init+0x437/0x520 [ 4768.066911] path_openat+0xf7/0x500 [ 4768.066915] ? do_raw_spin_unlock+0x75/0x190 [ 4768.066918] do_filp_open+0x99/0x140 [ 4768.066922] ? getname_flags+0x6e/0x330 [ 4768.066926] ? __check_object_size+0xff/0x256 [ 4768.066931] ? do_raw_spin_unlock+0x75/0x190 [ 4768.066933] ? _raw_spin_unlock+0x12/0x30 [ 4768.066936] do_sys_openat2+0x2b4/0x410 [ 4768.066939] do_sys_open+0x73/0xa0 [ 4768.066941] __x64_sys_openat+0x24/0x30 [ 4768.066944] do_syscall_64+0xc1/0x440 [ 4768.066948] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4768.066952] RIP: 0033:0x7f1a63983332 [ 4768.066958] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4768.066961] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4768.066965] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4768.066967] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4768.066968] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4768.066970] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4768.066971] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4768.611772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4768.649173] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4768.664312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4768.680593] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4768.683129] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4768.691127] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4768.694121] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4768.696587] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4768.699216] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4768.702263] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4768.705841] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4768.710443] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4768.714288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4768.717136] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4768.720308] Call Trace: [ 4768.721249] [ 4768.722039] ? show_regs.cold.9+0x22/0x2f [ 4768.723567] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4768.725418] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4768.727108] ? watchdog+0x40/0x40 [ 4768.728356] ? hrtimer_interrupt+0x128/0x2c0 [ 4768.729574] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4768.732626] ? apic_timer_interrupt+0xf/0x20 [ 4768.734599] [ 4768.736921] ? native_safe_halt+0x2e/0x50 [ 4768.738415] kvm_wait+0xcd/0xe0 [ 4768.739379] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4768.740963] do_raw_spin_lock+0x177/0x190 [ 4768.742300] _raw_spin_lock+0x21/0x30 [ 4768.744786] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4768.746531] ? do_raw_spin_unlock+0x75/0x190 [ 4768.748972] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4768.753080] ? do_wait_intr+0xf0/0xf0 [ 4768.756288] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4768.761600] mdc_close+0x53b/0xd70 [mdc] [ 4768.767365] lmv_close+0x18d/0x490 [lmv] [ 4768.769371] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4768.772723] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4768.778849] ll_md_close+0x274/0x800 [lustre] [ 4768.780652] ll_file_release+0x128/0xc50 [lustre] [ 4768.783325] __fput+0x10d/0x440 [ 4768.785220] ____fput+0x12/0x20 [ 4768.786884] task_work_run+0xc6/0x110 [ 4768.789135] exit_to_usermode_loop+0x1dd/0x1f0 [ 4768.791460] do_syscall_64+0x3d6/0x440 [ 4768.793253] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4768.795586] RIP: 0033:0x7f1a63983cd8 [ 4768.798521] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4768.812015] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4768.815253] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4768.819326] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4768.824526] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4768.830618] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4768.834108] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4772.837794] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2749s! [ 4772.842238] Showing busy workqueues and worker pools: [ 4772.844712] workqueue events: flags=0x0 [ 4772.847691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4772.850694] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4772.854283] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4772.856509] pending: sec_gc_main [ptlrpc] [ 4772.863504] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4772.870799] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4772.877572] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4772.881300] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4772.897824] workqueue events_long: flags=0x0 [ 4772.899911] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4772.904970] pending: lu_objects_flush [obdclass] [ 4772.910485] workqueue events_unbound: flags=0x2 [ 4772.911950] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4772.920235] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4772.931650] workqueue events_power_efficient: flags=0x80 [ 4772.933351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4772.946144] pending: neigh_periodic_work, do_cache_clean [ 4772.949912] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4772.953012] pending: neigh_periodic_work, check_lifetime [ 4772.957386] workqueue mm_percpu_wq: flags=0x8 [ 4772.960209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4772.963655] pending: vmstat_update [ 4772.965755] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4772.967668] pending: vmstat_update [ 4772.969147] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4772.970857] pending: vmstat_update [ 4772.971931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4772.976766] pending: vmstat_update [ 4772.978099] workqueue writeback: flags=0x4a [ 4772.980323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4772.985018] pending: wb_workfn, wb_workfn [ 4772.986930] workqueue xprtiod: flags=0xa [ 4772.987983] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4772.992354] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4772.997013] workqueue ipv6_addrconf: flags=0x40008 [ 4773.002119] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4773.004770] pending: addrconf_verify_work [ 4773.009025] workqueue knbd0-recv: flags=0x1a [ 4773.010996] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4773.013978] in-flight: 647:recv_work [ 4773.015838] workqueue ptlrpc_pinger: flags=0x2 [ 4773.019067] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4773.030989] pending: ptlrpc_pinger_main [ptlrpc] [ 4773.036879] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4763s workers=2 idle: 652 [ 4782.474018] rcu: INFO: rcu_sched self-detected stall on CPU [ 4782.476003] rcu: 3-...!: (1662456 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 4782.478896] (t=2760072 jiffies g=269633 q=1119) [ 4782.480289] rcu: rcu_sched kthread starved for 2759720 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 4782.483637] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4782.486795] rcu: RCU grace-period kthread stack dump: [ 4782.488229] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4782.490935] Call Trace: [ 4782.491529] __schedule+0x351/0xcb0 [ 4782.492491] schedule+0xc0/0x180 [ 4782.493468] schedule_timeout+0xb4/0x190 [ 4782.494654] ? do_raw_spin_unlock+0x75/0x190 [ 4782.495938] ? __next_timer_interrupt+0x160/0x160 [ 4782.497247] ? prepare_to_swait_event+0xf3/0x140 [ 4782.498480] rcu_gp_kthread+0x8dd/0x15b0 [ 4782.499943] ? force_qs_rnp+0x2c0/0x2c0 [ 4782.501055] kthread+0x1d1/0x200 [ 4782.501948] ? set_kthread_struct+0x70/0x70 [ 4782.503141] ret_from_fork+0x1f/0x30 [ 4782.504600] rcu: Stack dump where RCU GP kthread last ran: [ 4782.507309] Sending NMI from CPU 3 to CPUs 0: [ 4782.508526] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 4782.509488] NMI backtrace for cpu 3 [ 4782.511998] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4782.518275] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4782.522429] Call Trace: [ 4782.523509] [ 4782.524810] ? dump_stack+0xbb/0x10e [ 4782.526324] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4782.527934] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4782.529587] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4782.531158] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4782.533727] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4782.535363] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4782.537497] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4782.539467] ? update_process_times+0x6d/0xc0 [ 4782.541336] ? tick_sched_handle.isra.18+0x27/0x90 [ 4782.543300] ? tick_sched_timer+0x53/0xb0 [ 4782.544916] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4782.546439] ? tick_sched_handle.isra.18+0x90/0x90 [ 4782.548739] ? hrtimer_interrupt+0x128/0x2c0 [ 4782.550462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4782.552395] ? apic_timer_interrupt+0xf/0x20 [ 4782.556851] [ 4782.557843] ? native_safe_halt+0x2e/0x50 [ 4782.559705] ? kvm_wait+0xcd/0xe0 [ 4782.561122] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4782.564080] ? do_raw_spin_lock+0x177/0x190 [ 4782.566981] ? _raw_spin_lock+0x21/0x30 [ 4782.571719] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4782.576275] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4782.579652] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4782.582126] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4782.585944] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4782.590175] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4782.592148] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4782.602401] ? __next_timer_interrupt+0x160/0x160 [ 4782.605006] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4782.609171] ? do_wait_intr+0xf0/0xf0 [ 4782.613221] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4782.617839] ? kthread+0x1d1/0x200 [ 4782.620140] ? set_kthread_struct+0x70/0x70 [ 4782.623176] ? ret_from_fork+0x1f/0x30 [ 4788.057031] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 4788.062131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4788.113089] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4788.121229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4788.125735] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4788.128584] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4788.137442] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4788.141222] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4788.144972] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4788.151915] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4788.155895] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4788.163240] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4788.166960] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4788.173248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4788.178296] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4788.186944] Call Trace: [ 4788.189013] [ 4788.195953] ? show_regs.cold.9+0x22/0x2f [ 4788.201577] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4788.215216] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4788.220158] ? watchdog+0x40/0x40 [ 4788.223832] ? hrtimer_interrupt+0x128/0x2c0 [ 4788.226820] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4788.228297] ? apic_timer_interrupt+0xf/0x20 [ 4788.231418] [ 4788.232179] ? native_safe_halt+0x2e/0x50 [ 4788.233861] kvm_wait+0xcd/0xe0 [ 4788.235187] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4788.237068] do_raw_spin_lock+0x177/0x190 [ 4788.238392] _raw_spin_lock+0x21/0x30 [ 4788.241601] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4788.244031] ? do_raw_spin_unlock+0x75/0x190 [ 4788.246724] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4788.248765] ? do_wait_intr+0xf0/0xf0 [ 4788.250238] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4788.252236] mdc_close+0x53b/0xd70 [mdc] [ 4788.253696] lmv_close+0x18d/0x490 [lmv] [ 4788.255123] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4788.257446] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4788.259139] ll_md_close+0x274/0x800 [lustre] [ 4788.260849] ll_file_release+0x128/0xc50 [lustre] [ 4788.262714] __fput+0x10d/0x440 [ 4788.263786] ____fput+0x12/0x20 [ 4788.264866] task_work_run+0xc6/0x110 [ 4788.266065] exit_to_usermode_loop+0x1dd/0x1f0 [ 4788.267601] do_syscall_64+0x3d6/0x440 [ 4788.269027] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4788.270743] RIP: 0033:0x7f1a63983cd8 [ 4788.271814] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4788.277936] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4788.281182] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4788.284752] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4788.288620] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4788.292626] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4788.296716] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4796.057711] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4796.061215] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4796.095771] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4796.103380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4796.108765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.115255] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4796.124775] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4796.128254] RAX: 000000000000314c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4796.133479] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4796.138659] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4796.144048] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4796.152707] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4796.163491] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4796.168589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4796.172938] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4796.177714] Call Trace: [ 4796.179006] [ 4796.180268] ? show_regs.cold.9+0x22/0x2f [ 4796.184568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4796.188234] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4796.193618] ? watchdog+0x40/0x40 [ 4796.195725] ? hrtimer_interrupt+0x128/0x2c0 [ 4796.199507] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4796.202553] ? apic_timer_interrupt+0xf/0x20 [ 4796.205729] [ 4796.208581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.217514] do_raw_spin_lock+0x177/0x190 [ 4796.224960] _raw_spin_lock+0x21/0x30 [ 4796.228222] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4796.234034] ? do_raw_spin_unlock+0x75/0x190 [ 4796.239245] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4796.241968] ? do_wait_intr+0xf0/0xf0 [ 4796.245965] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4796.251780] mdc_close+0x53b/0xd70 [mdc] [ 4796.253764] lmv_close+0x18d/0x490 [lmv] [ 4796.255886] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4796.258753] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4796.261009] ll_md_close+0x274/0x800 [lustre] [ 4796.263668] ll_file_release+0x128/0xc50 [lustre] [ 4796.265882] __fput+0x10d/0x440 [ 4796.267590] ____fput+0x12/0x20 [ 4796.269041] task_work_run+0xc6/0x110 [ 4796.270991] exit_to_usermode_loop+0x1dd/0x1f0 [ 4796.273196] do_syscall_64+0x3d6/0x440 [ 4796.274946] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4796.277691] RIP: 0033:0x7f1a63983cd8 [ 4796.279573] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4796.288727] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4796.292389] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4796.295229] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4796.298651] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4796.302237] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4796.309844] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4803.047028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2780s! [ 4803.059228] Showing busy workqueues and worker pools: [ 4803.063401] workqueue events: flags=0x0 [ 4803.065994] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4803.069352] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4803.075545] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4803.081163] pending: sec_gc_main [ptlrpc] [ 4803.083832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4803.088954] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4803.095664] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4803.098887] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4803.105476] workqueue events_long: flags=0x0 [ 4803.108719] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4803.112042] pending: lu_objects_flush [obdclass] [ 4803.116753] workqueue events_unbound: flags=0x2 [ 4803.118068] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4803.123522] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4803.130970] workqueue events_power_efficient: flags=0x80 [ 4803.134331] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4803.140100] pending: neigh_periodic_work, do_cache_clean [ 4803.143919] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4803.151218] pending: neigh_periodic_work, check_lifetime [ 4803.155801] workqueue mm_percpu_wq: flags=0x8 [ 4803.161289] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4803.165452] pending: vmstat_update [ 4803.167009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4803.171098] pending: vmstat_update [ 4803.174424] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4803.181531] pending: vmstat_update [ 4803.182637] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4803.187962] pending: vmstat_update [ 4803.192925] workqueue writeback: flags=0x4a [ 4803.195315] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4803.200291] pending: wb_workfn, wb_workfn [ 4803.204908] workqueue xprtiod: flags=0xa [ 4803.208362] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4803.211301] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4803.217846] workqueue ipv6_addrconf: flags=0x40008 [ 4803.219994] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4803.222865] pending: addrconf_verify_work [ 4803.224798] workqueue knbd0-recv: flags=0x1a [ 4803.227526] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4803.230157] in-flight: 647:recv_work [ 4803.232992] workqueue ptlrpc_pinger: flags=0x2 [ 4803.235555] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4803.237781] pending: ptlrpc_pinger_main [ptlrpc] [ 4803.242565] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4793s workers=2 idle: 652 [ 4808.061906] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4808.067851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4808.104236] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4808.110762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4808.115160] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4808.117882] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4808.126626] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4808.129881] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4808.132801] RDX: 0000000000000997 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4808.135662] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4808.139401] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4808.142391] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4808.145341] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4808.148843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4808.151191] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4808.153782] Call Trace: [ 4808.155335] [ 4808.156085] ? show_regs.cold.9+0x22/0x2f [ 4808.157807] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4808.159739] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4808.161489] ? watchdog+0x40/0x40 [ 4808.162764] ? hrtimer_interrupt+0x128/0x2c0 [ 4808.164493] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4808.166402] ? apic_timer_interrupt+0xf/0x20 [ 4808.168212] [ 4808.169063] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4808.171190] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4808.174876] do_raw_spin_lock+0x177/0x190 [ 4808.180228] _raw_spin_lock+0x21/0x30 [ 4808.181904] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4808.184497] obd_mod_free+0x46/0x140 [ptlrpc] [ 4808.186837] mdc_commit_open+0x91/0xb0 [mdc] [ 4808.189081] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4808.191431] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4808.193981] after_reply+0xe2c/0x1780 [ptlrpc] [ 4808.196047] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4808.198226] ? __next_timer_interrupt+0x160/0x160 [ 4808.200139] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4808.201953] ? do_wait_intr+0xf0/0xf0 [ 4808.203177] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4808.205498] kthread+0x1d1/0x200 [ 4808.206787] ? set_kthread_struct+0x70/0x70 [ 4808.208491] ret_from_fork+0x1f/0x30 [ 4816.057452] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 4816.066277] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4816.152642] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4816.168487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4816.184235] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4816.191841] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4816.224739] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4816.243488] RAX: 0000000000002be7 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4816.249899] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4816.258300] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4816.268394] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4816.280194] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4816.295498] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4816.317639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4816.329061] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4816.338513] Call Trace: [ 4816.339439] [ 4816.340196] ? show_regs.cold.9+0x22/0x2f [ 4816.357432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4816.366416] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4816.377883] ? watchdog+0x40/0x40 [ 4816.398839] ? hrtimer_interrupt+0x128/0x2c0 [ 4816.400594] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4816.415913] ? apic_timer_interrupt+0xf/0x20 [ 4816.420477] [ 4816.429770] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4816.438946] do_raw_spin_lock+0x177/0x190 [ 4816.451757] _raw_spin_lock+0x21/0x30 [ 4816.457756] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4816.469687] ? do_raw_spin_unlock+0x75/0x190 [ 4816.477258] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4816.483283] ? do_wait_intr+0xf0/0xf0 [ 4816.491354] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4816.502055] mdc_close+0x53b/0xd70 [mdc] [ 4816.505693] lmv_close+0x18d/0x490 [lmv] [ 4816.508883] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4816.517997] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4816.530359] ll_md_close+0x274/0x800 [lustre] [ 4816.539605] ll_file_release+0x128/0xc50 [lustre] [ 4816.546277] __fput+0x10d/0x440 [ 4816.552680] ____fput+0x12/0x20 [ 4816.563132] task_work_run+0xc6/0x110 [ 4816.574875] exit_to_usermode_loop+0x1dd/0x1f0 [ 4816.580707] do_syscall_64+0x3d6/0x440 [ 4816.590901] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4816.610006] RIP: 0033:0x7f1a63983cd8 [ 4816.626653] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4816.651521] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4816.668073] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4816.674141] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4816.684392] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4816.696430] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4816.712243] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4824.056345] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4824.068711] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4824.143878] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4824.160892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4824.171821] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4824.178281] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4824.201390] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4824.215360] RAX: 000000000000708a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4824.229734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4824.247231] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4824.257894] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4824.261183] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4824.264744] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4824.271927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4824.279820] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4824.294559] Call Trace: [ 4824.299511] [ 4824.300698] ? show_regs.cold.9+0x22/0x2f [ 4824.308018] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4824.319398] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4824.320738] ? watchdog+0x40/0x40 [ 4824.330794] ? hrtimer_interrupt+0x128/0x2c0 [ 4824.337007] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4824.343065] ? apic_timer_interrupt+0xf/0x20 [ 4824.353248] [ 4824.355719] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4824.365950] do_raw_spin_lock+0x177/0x190 [ 4824.372665] _raw_spin_lock+0x21/0x30 [ 4824.378555] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4824.388266] ? do_raw_spin_unlock+0x75/0x190 [ 4824.395063] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4824.410821] ? do_wait_intr+0xf0/0xf0 [ 4824.415575] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4824.427006] mdc_close+0x53b/0xd70 [mdc] [ 4824.432111] lmv_close+0x18d/0x490 [lmv] [ 4824.445778] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4824.458997] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4824.464346] ll_md_close+0x274/0x800 [lustre] [ 4824.474976] ll_file_release+0x128/0xc50 [lustre] [ 4824.486883] __fput+0x10d/0x440 [ 4824.488986] ____fput+0x12/0x20 [ 4824.489987] task_work_run+0xc6/0x110 [ 4824.501106] exit_to_usermode_loop+0x1dd/0x1f0 [ 4824.505478] do_syscall_64+0x3d6/0x440 [ 4824.513694] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4824.517884] RIP: 0033:0x7f1a63983cd8 [ 4824.520506] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4824.543878] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4824.551676] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4824.565027] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4824.580403] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4824.587616] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4824.607348] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4828.053428] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4828.072196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4828.133897] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4828.179888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4828.205262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4828.221142] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4828.260912] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4828.279227] RAX: 0000000000006e86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4828.290838] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4828.313021] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4828.333864] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4828.358070] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4828.378218] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4828.390506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4828.404535] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4828.418824] Call Trace: [ 4828.421996] [ 4828.428210] ? show_regs.cold.9+0x22/0x2f [ 4828.441935] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4828.443633] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4828.457083] ? watchdog+0x40/0x40 [ 4828.466985] ? hrtimer_interrupt+0x128/0x2c0 [ 4828.476438] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4828.477922] ? apic_timer_interrupt+0xf/0x20 [ 4828.492867] [ 4828.493910] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4828.502808] do_raw_spin_lock+0x177/0x190 [ 4828.504249] _raw_spin_lock+0x21/0x30 [ 4828.517925] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4828.524526] ? do_raw_spin_unlock+0x75/0x190 [ 4828.526041] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4828.541975] ? do_wait_intr+0xf0/0xf0 [ 4828.558750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4828.581285] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4828.592061] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4828.594061] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4828.616264] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4828.620346] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4828.637113] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4828.638809] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4828.653678] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4828.670444] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4828.672192] lmv_intent_open+0x338/0x1150 [lmv] [ 4828.694036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4828.695990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4828.709186] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4828.718780] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4828.746477] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4828.769166] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4828.791896] ? do_raw_spin_unlock+0x75/0x190 [ 4828.793406] ? _raw_spin_unlock+0x12/0x30 [ 4828.794871] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4828.824102] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4828.833816] ll_atomic_open+0x36a/0xd70 [lustre] [ 4828.848485] lookup_open+0x356/0xb40 [ 4828.861364] do_last+0x313/0xfc0 [ 4828.863072] ? nd_jump_root+0xe5/0x160 [ 4828.865472] ? path_init+0x437/0x520 [ 4828.867603] path_openat+0xf7/0x500 [ 4828.869695] ? do_raw_spin_unlock+0x75/0x190 [ 4828.872206] do_filp_open+0x99/0x140 [ 4828.874607] ? getname_flags+0x6e/0x330 [ 4828.877811] ? __check_object_size+0xff/0x256 [ 4828.881834] ? do_raw_spin_unlock+0x75/0x190 [ 4828.890408] ? _raw_spin_unlock+0x12/0x30 [ 4828.896102] do_sys_openat2+0x2b4/0x410 [ 4828.899230] do_sys_open+0x73/0xa0 [ 4828.901943] __x64_sys_openat+0x24/0x30 [ 4828.904266] do_syscall_64+0xc1/0x440 [ 4828.912781] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4828.916992] RIP: 0033:0x7f1a63983332 [ 4828.924932] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4828.940285] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4828.950452] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4828.957060] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4828.964849] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4828.969284] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4828.978704] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4833.248684] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2810s! [ 4833.258142] Showing busy workqueues and worker pools: [ 4833.264544] workqueue events: flags=0x0 [ 4833.266214] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4833.273580] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4833.282192] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4833.286795] pending: sec_gc_main [ptlrpc] [ 4833.292324] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4833.297283] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4833.308421] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4833.314544] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4833.327420] workqueue events_long: flags=0x0 [ 4833.330489] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4833.337784] pending: lu_objects_flush [obdclass] [ 4833.343767] workqueue events_unbound: flags=0x2 [ 4833.348758] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4833.355301] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4833.366140] workqueue events_power_efficient: flags=0x80 [ 4833.373348] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4833.379398] pending: neigh_periodic_work, do_cache_clean [ 4833.383208] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4833.391248] pending: neigh_periodic_work, check_lifetime [ 4833.396545] workqueue mm_percpu_wq: flags=0x8 [ 4833.402058] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4833.408641] pending: vmstat_update [ 4833.410809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4833.414027] pending: vmstat_update [ 4833.415876] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4833.419073] pending: vmstat_update [ 4833.420506] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4833.423848] pending: vmstat_update [ 4833.425743] workqueue writeback: flags=0x4a [ 4833.427921] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4833.430739] pending: wb_workfn, wb_workfn [ 4833.433172] workqueue xprtiod: flags=0xa [ 4833.434785] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4833.438016] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4833.441387] workqueue ipv6_addrconf: flags=0x40008 [ 4833.443363] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4833.445820] pending: addrconf_verify_work [ 4833.447961] workqueue knbd0-recv: flags=0x1a [ 4833.449588] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4833.452656] in-flight: 647:recv_work [ 4833.454743] workqueue ptlrpc_pinger: flags=0x2 [ 4833.456769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4833.459490] pending: ptlrpc_pinger_main [ptlrpc] [ 4833.465281] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4823s workers=2 idle: 652 [ 4836.058551] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4836.064857] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4836.092244] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4836.099189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4836.105264] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4836.107373] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4836.119200] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4836.122918] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4836.127592] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4836.131865] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4836.136513] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4836.140526] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4836.144252] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4836.151711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4836.154823] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4836.160079] Call Trace: [ 4836.161827] [ 4836.162930] ? show_regs.cold.9+0x22/0x2f [ 4836.166211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4836.168375] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4836.170892] ? watchdog+0x40/0x40 [ 4836.172499] ? hrtimer_interrupt+0x128/0x2c0 [ 4836.174416] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4836.178974] ? apic_timer_interrupt+0xf/0x20 [ 4836.180761] [ 4836.181723] ? native_safe_halt+0x2e/0x50 [ 4836.183713] kvm_wait+0xcd/0xe0 [ 4836.185415] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4836.187873] do_raw_spin_lock+0x177/0x190 [ 4836.190344] _raw_spin_lock+0x21/0x30 [ 4836.192551] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4836.197113] obd_mod_free+0x46/0x140 [ptlrpc] [ 4836.199601] mdc_commit_open+0x91/0xb0 [mdc] [ 4836.201466] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4836.206572] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4836.211314] after_reply+0xe2c/0x1780 [ptlrpc] [ 4836.214043] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4836.218420] ? __next_timer_interrupt+0x160/0x160 [ 4836.220802] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4836.224362] ? do_wait_intr+0xf0/0xf0 [ 4836.225872] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4836.228864] kthread+0x1d1/0x200 [ 4836.230569] ? set_kthread_struct+0x70/0x70 [ 4836.233607] ret_from_fork+0x1f/0x30 [ 4844.057044] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4844.060615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4844.100439] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4844.107474] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4844.112087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4844.115446] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4844.134999] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4844.138795] RAX: 000000000000028a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4844.145022] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4844.151744] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4844.155585] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4844.159148] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4844.163236] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4844.167520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4844.169636] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4844.172205] Call Trace: [ 4844.173468] [ 4844.174330] ? show_regs.cold.9+0x22/0x2f [ 4844.176262] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4844.178688] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4844.189164] ? watchdog+0x40/0x40 [ 4844.190674] ? hrtimer_interrupt+0x128/0x2c0 [ 4844.196803] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4844.200924] ? apic_timer_interrupt+0xf/0x20 [ 4844.206364] [ 4844.207181] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4844.214451] do_raw_spin_lock+0x177/0x190 [ 4844.216245] _raw_spin_lock+0x21/0x30 [ 4844.221353] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4844.228156] ? do_raw_spin_unlock+0x75/0x190 [ 4844.230107] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4844.234817] ? do_wait_intr+0xf0/0xf0 [ 4844.238711] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4844.241384] mdc_close+0x53b/0xd70 [mdc] [ 4844.243556] lmv_close+0x18d/0x490 [lmv] [ 4844.245873] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4844.251309] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4844.254318] ll_md_close+0x274/0x800 [lustre] [ 4844.257021] ll_file_release+0x128/0xc50 [lustre] [ 4844.259403] __fput+0x10d/0x440 [ 4844.261028] ____fput+0x12/0x20 [ 4844.262507] task_work_run+0xc6/0x110 [ 4844.264271] exit_to_usermode_loop+0x1dd/0x1f0 [ 4844.266254] do_syscall_64+0x3d6/0x440 [ 4844.268244] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4844.271299] RIP: 0033:0x7f1a63983cd8 [ 4844.273074] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4844.282103] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4844.286106] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4844.289794] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4844.293169] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4844.296559] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4844.299312] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4852.056068] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4852.061971] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4852.095361] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4852.103535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4852.109854] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4852.114796] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4852.128709] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4852.133850] RAX: 0000000000004711 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4852.141018] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4852.147922] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4852.153745] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4852.159292] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4852.166036] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4852.172061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4852.185180] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4852.191988] Call Trace: [ 4852.205192] [ 4852.210125] ? show_regs.cold.9+0x22/0x2f [ 4852.212292] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4852.220535] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4852.230156] ? watchdog+0x40/0x40 [ 4852.231712] ? hrtimer_interrupt+0x128/0x2c0 [ 4852.240425] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4852.247190] ? apic_timer_interrupt+0xf/0x20 [ 4852.254260] [ 4852.258283] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4852.263698] do_raw_spin_lock+0x177/0x190 [ 4852.273298] _raw_spin_lock+0x21/0x30 [ 4852.280832] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4852.286042] ? do_raw_spin_unlock+0x75/0x190 [ 4852.292218] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4852.301300] ? do_wait_intr+0xf0/0xf0 [ 4852.304927] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4852.309163] mdc_close+0x53b/0xd70 [mdc] [ 4852.313146] lmv_close+0x18d/0x490 [lmv] [ 4852.315757] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4852.319158] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4852.322645] ll_md_close+0x274/0x800 [lustre] [ 4852.325834] ll_file_release+0x128/0xc50 [lustre] [ 4852.330218] __fput+0x10d/0x440 [ 4852.332188] ____fput+0x12/0x20 [ 4852.334475] task_work_run+0xc6/0x110 [ 4852.336596] exit_to_usermode_loop+0x1dd/0x1f0 [ 4852.339296] do_syscall_64+0x3d6/0x440 [ 4852.346078] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4852.352204] RIP: 0033:0x7f1a63983cd8 [ 4852.355518] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4852.368120] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4852.372810] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4852.377807] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4852.387478] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4852.394267] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4852.402471] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4860.053062] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 4860.056356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4860.091137] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4860.097971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4860.102282] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4860.104633] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4860.114294] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4860.118098] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4860.120937] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4860.125292] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4860.127773] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4860.132245] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4860.136059] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4860.140435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4860.143390] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4860.147085] Call Trace: [ 4860.148409] [ 4860.149588] ? show_regs.cold.9+0x22/0x2f [ 4860.152151] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4860.155753] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4860.158479] ? watchdog+0x40/0x40 [ 4860.160188] ? hrtimer_interrupt+0x128/0x2c0 [ 4860.162144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4860.167613] ? apic_timer_interrupt+0xf/0x20 [ 4860.171658] [ 4860.172920] ? native_safe_halt+0x2e/0x50 [ 4860.183374] kvm_wait+0xcd/0xe0 [ 4860.191452] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4860.198918] do_raw_spin_lock+0x177/0x190 [ 4860.204224] _raw_spin_lock+0x21/0x30 [ 4860.208675] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4860.213652] ? do_raw_spin_unlock+0x75/0x190 [ 4860.217557] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4860.221043] ? do_wait_intr+0xf0/0xf0 [ 4860.222941] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4860.231755] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4860.236032] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4860.244109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.249068] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4860.252977] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4860.255758] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4860.260914] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.266388] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4860.276535] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4860.282145] lmv_intent_open+0x338/0x1150 [lmv] [ 4860.286462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.291953] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.295894] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4860.297307] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4860.303643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.308432] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4860.311498] ? do_raw_spin_unlock+0x75/0x190 [ 4860.313945] ? _raw_spin_unlock+0x12/0x30 [ 4860.317094] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4860.321175] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4860.325050] ll_atomic_open+0x36a/0xd70 [lustre] [ 4860.330082] lookup_open+0x356/0xb40 [ 4860.333523] do_last+0x313/0xfc0 [ 4860.337387] ? nd_jump_root+0xe5/0x160 [ 4860.343464] ? path_init+0x437/0x520 [ 4860.346447] path_openat+0xf7/0x500 [ 4860.349051] ? do_raw_spin_unlock+0x75/0x190 [ 4860.352100] do_filp_open+0x99/0x140 [ 4860.356632] ? getname_flags+0x6e/0x330 [ 4860.361474] ? __check_object_size+0xff/0x256 [ 4860.364299] ? do_raw_spin_unlock+0x75/0x190 [ 4860.366008] ? _raw_spin_unlock+0x12/0x30 [ 4860.368208] do_sys_openat2+0x2b4/0x410 [ 4860.370324] do_sys_open+0x73/0xa0 [ 4860.371580] __x64_sys_openat+0x24/0x30 [ 4860.372677] do_syscall_64+0xc1/0x440 [ 4860.374927] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4860.377534] RIP: 0033:0x7f1a63983332 [ 4860.379945] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4860.398263] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4860.407024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4860.413304] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4860.420052] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4860.424905] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4860.432299] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4863.968050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2840s! [ 4863.970939] Showing busy workqueues and worker pools: [ 4863.986962] workqueue events: flags=0x0 [ 4863.988419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4864.001442] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4864.010522] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4864.014721] pending: sec_gc_main [ptlrpc] [ 4864.016639] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4864.024322] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4864.035716] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4864.051027] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4864.058054] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4864.063681] workqueue events_long: flags=0x0 [ 4864.066458] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4864.066938] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4864.066945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4864.066947] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4864.066960] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4864.066964] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4864.066968] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4864.066970] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4864.066972] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4864.066975] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4864.066977] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4864.066980] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4864.066982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4864.066984] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4864.066992] Call Trace: [ 4864.067006] [ 4864.067013] ? show_regs.cold.9+0x22/0x2f [ 4864.067023] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4864.067031] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4864.067036] ? watchdog+0x40/0x40 [ 4864.067041] ? hrtimer_interrupt+0x128/0x2c0 [ 4864.067045] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4864.067053] ? apic_timer_interrupt+0xf/0x20 [ 4864.067058] [ 4864.067062] ? native_safe_halt+0x2e/0x50 [ 4864.067071] kvm_wait+0xcd/0xe0 [ 4864.067079] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4864.067085] do_raw_spin_lock+0x177/0x190 [ 4864.067089] _raw_spin_lock+0x21/0x30 [ 4864.067094] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4864.067421] obd_mod_free+0x46/0x140 [ptlrpc] [ 4864.067934] mdc_commit_open+0x91/0xb0 [mdc] [ 4864.067976] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4864.072261] pwq 2: [ 4864.129314] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4864.129595] after_reply+0xe2c/0x1780 [ptlrpc] [ 4864.142368] cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4864.142389] pending: lu_objects_flush [obdclass] [ 4864.142582] workqueue events_unbound: flags=0x2 [ 4864.142585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4864.142592] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4864.142618] workqueue events_power_efficient: flags=0x80 [ 4864.142620] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4864.142628] pending: neigh_periodic_work, do_cache_clean [ 4864.142644] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4864.142650] pending: neigh_periodic_work, check_lifetime [ 4864.142661] workqueue mm_percpu_wq: flags=0x8 [ 4864.142663] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4864.142670] pending: vmstat_update [ 4864.142676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4864.142683] pending: vmstat_update [ 4864.142687] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4864.142693] pending: vmstat_update [ 4864.142696] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4864.142703] pending: vmstat_update [ 4864.142710] workqueue writeback: flags=0x4a [ 4864.142712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4864.142718] pending: wb_workfn, wb_workfn [ 4864.142733] workqueue xprtiod: flags=0xa [ 4864.142735] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4864.154657] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4864.164604] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4864.164661] workqueue ipv6_addrconf: flags=0x40008 [ 4864.164667] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4864.164679] pending: addrconf_verify_work [ 4864.164694] workqueue knbd0-recv: flags=0x1a [ 4864.164696] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4864.164704] in-flight: 647:recv_work [ 4864.164716] workqueue ptlrpc_pinger: flags=0x2 [ 4864.164718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4864.164724] pending: ptlrpc_pinger_main [ptlrpc] [ 4864.165068] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4854s workers=2 idle: 652 [ 4864.544199] ? __next_timer_interrupt+0x160/0x160 [ 4864.551197] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4864.557304] ? do_wait_intr+0xf0/0xf0 [ 4864.560610] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4864.568528] kthread+0x1d1/0x200 [ 4864.570871] ? set_kthread_struct+0x70/0x70 [ 4864.575128] ret_from_fork+0x1f/0x30 [ 4872.057059] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4872.061541] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4872.123653] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4872.140406] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4872.159538] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4872.170012] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4872.203416] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4872.215736] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4872.226390] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4872.235618] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4872.242587] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4872.251357] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4872.262953] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4872.269589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4872.279850] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4872.293355] Call Trace: [ 4872.295918] [ 4872.299742] ? show_regs.cold.9+0x22/0x2f [ 4872.308351] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4872.317033] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4872.318774] ? watchdog+0x40/0x40 [ 4872.334625] ? hrtimer_interrupt+0x128/0x2c0 [ 4872.338818] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4872.343985] ? apic_timer_interrupt+0xf/0x20 [ 4872.353619] [ 4872.355461] ? native_safe_halt+0x2e/0x50 [ 4872.363146] kvm_wait+0xcd/0xe0 [ 4872.367206] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4872.372362] do_raw_spin_lock+0x177/0x190 [ 4872.377405] _raw_spin_lock+0x21/0x30 [ 4872.378754] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4872.381594] ? do_raw_spin_unlock+0x75/0x190 [ 4872.383844] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4872.388393] ? do_wait_intr+0xf0/0xf0 [ 4872.390387] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4872.394559] mdc_close+0x53b/0xd70 [mdc] [ 4872.399780] lmv_close+0x18d/0x490 [lmv] [ 4872.401375] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4872.407029] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4872.410204] ll_md_close+0x274/0x800 [lustre] [ 4872.413954] ll_file_release+0x128/0xc50 [lustre] [ 4872.418118] __fput+0x10d/0x440 [ 4872.423216] ____fput+0x12/0x20 [ 4872.425326] task_work_run+0xc6/0x110 [ 4872.434605] exit_to_usermode_loop+0x1dd/0x1f0 [ 4872.441875] do_syscall_64+0x3d6/0x440 [ 4872.450213] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4872.456402] RIP: 0033:0x7f1a63983cd8 [ 4872.463090] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4872.480562] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4872.484573] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4872.495572] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4872.503228] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4872.524473] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4872.541787] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4880.057861] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4880.067180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4880.151878] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4880.167044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4880.172267] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4880.180300] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4880.213158] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4880.232465] RAX: 0000000000005584 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4880.249881] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4880.257641] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4880.262491] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4880.278252] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4880.293141] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4880.304045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4880.308038] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4880.315687] Call Trace: [ 4880.318315] [ 4880.320554] ? show_regs.cold.9+0x22/0x2f [ 4880.324919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4880.332212] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4880.338288] ? watchdog+0x40/0x40 [ 4880.339466] ? hrtimer_interrupt+0x128/0x2c0 [ 4880.345367] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4880.353668] ? apic_timer_interrupt+0xf/0x20 [ 4880.361936] [ 4880.365290] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4880.367278] do_raw_spin_lock+0x177/0x190 [ 4880.378852] _raw_spin_lock+0x21/0x30 [ 4880.383507] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4880.390927] ? do_raw_spin_unlock+0x75/0x190 [ 4880.392458] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4880.404232] ? do_wait_intr+0xf0/0xf0 [ 4880.409161] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4880.418244] mdc_close+0x53b/0xd70 [mdc] [ 4880.421118] lmv_close+0x18d/0x490 [lmv] [ 4880.423181] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4880.426684] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4880.433189] ll_md_close+0x274/0x800 [lustre] [ 4880.439090] ll_file_release+0x128/0xc50 [lustre] [ 4880.443478] __fput+0x10d/0x440 [ 4880.454040] ____fput+0x12/0x20 [ 4880.456368] task_work_run+0xc6/0x110 [ 4880.464625] exit_to_usermode_loop+0x1dd/0x1f0 [ 4880.467381] do_syscall_64+0x3d6/0x440 [ 4880.470495] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4880.472794] RIP: 0033:0x7f1a63983cd8 [ 4880.474201] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4880.486251] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4880.491130] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4880.497422] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4880.504375] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4880.517316] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4880.522197] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4892.053025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 4892.058584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4892.061456] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4892.133649] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4892.133664] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4892.133667] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4892.133682] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4892.133686] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4892.133693] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4892.133696] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4892.133698] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4892.133701] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4892.133703] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4892.133706] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4892.133710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4892.133712] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4892.133722] Call Trace: [ 4892.133728] [ 4892.133735] ? show_regs.cold.9+0x22/0x2f [ 4892.137268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4892.137371] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4892.137379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4892.137382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4892.137395] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4892.137399] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4892.137404] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4892.137406] RDX: 0000000000000d5e RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4892.137409] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4892.137411] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4892.137413] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4892.137420] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4892.137424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4892.137426] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4892.137433] Call Trace: [ 4892.137438] [ 4892.137444] ? show_regs.cold.9+0x22/0x2f [ 4892.137453] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4892.137459] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4892.137464] ? watchdog+0x40/0x40 [ 4892.137542] ? hrtimer_interrupt+0x128/0x2c0 [ 4892.137581] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4892.137644] ? apic_timer_interrupt+0xf/0x20 [ 4892.137714] [ 4892.137718] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4892.137723] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4892.137727] do_raw_spin_lock+0x177/0x190 [ 4892.137731] _raw_spin_lock+0x21/0x30 [ 4892.137737] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4892.138094] obd_mod_free+0x46/0x140 [ptlrpc] [ 4892.138405] mdc_commit_open+0x91/0xb0 [mdc] [ 4892.138444] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4892.138736] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4892.139142] after_reply+0xe2c/0x1780 [ptlrpc] [ 4892.139366] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4892.139747] ? __next_timer_interrupt+0x160/0x160 [ 4892.139756] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4892.148881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4892.148900] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4892.148904] ? watchdog+0x40/0x40 [ 4892.148914] ? hrtimer_interrupt+0x128/0x2c0 [ 4892.148918] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4892.148926] ? apic_timer_interrupt+0xf/0x20 [ 4892.148931] [ 4892.148935] ? native_safe_halt+0x2e/0x50 [ 4892.148939] kvm_wait+0xcd/0xe0 [ 4892.148952] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4892.148963] do_raw_spin_lock+0x177/0x190 [ 4892.148967] _raw_spin_lock+0x21/0x30 [ 4892.148975] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4892.149262] ? do_raw_spin_unlock+0x75/0x190 [ 4892.149268] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4892.149488] ? do_wait_intr+0xf0/0xf0 [ 4892.149499] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4892.149706] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4892.149922] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4892.150521] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4892.150665] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4892.150718] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4892.150750] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4892.150842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4892.150988] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4892.160078] ? do_wait_intr+0xf0/0xf0 [ 4892.160099] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4892.163365] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4892.163424] lmv_intent_open+0x338/0x1150 [lmv] [ 4892.163460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4892.163600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4892.163889] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4892.163917] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4892.164016] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4892.164113] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4892.164209] ? do_raw_spin_unlock+0x75/0x190 [ 4892.164218] ? _raw_spin_unlock+0x12/0x30 [ 4892.164225] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4892.164257] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4892.164279] ll_atomic_open+0x36a/0xd70 [lustre] [ 4892.164377] lookup_open+0x356/0xb40 [ 4892.164388] do_last+0x313/0xfc0 [ 4892.164392] ? nd_jump_root+0xe5/0x160 [ 4892.164397] ? path_init+0x437/0x520 [ 4892.164402] path_openat+0xf7/0x500 [ 4892.164407] ? do_raw_spin_unlock+0x75/0x190 [ 4892.164412] do_filp_open+0x99/0x140 [ 4892.164418] ? getname_flags+0x6e/0x330 [ 4892.164423] ? __check_object_size+0xff/0x256 [ 4892.164431] ? do_raw_spin_unlock+0x75/0x190 [ 4892.164435] ? _raw_spin_unlock+0x12/0x30 [ 4892.164439] do_sys_openat2+0x2b4/0x410 [ 4892.164444] do_sys_open+0x73/0xa0 [ 4892.164448] __x64_sys_openat+0x24/0x30 [ 4892.164452] do_syscall_64+0xc1/0x440 [ 4892.164459] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4892.164465] RIP: 0033:0x7f1a63983332 [ 4892.164787] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4892.164792] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4892.164797] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4892.164799] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4892.164801] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4892.164803] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4892.164805] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4892.590793] kthread+0x1d1/0x200 [ 4892.592833] ? set_kthread_struct+0x70/0x70 [ 4892.595087] ret_from_fork+0x1f/0x30 [ 4894.179850] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2871s! [ 4894.184708] Showing busy workqueues and worker pools: [ 4894.186633] workqueue events: flags=0x0 [ 4894.190982] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4894.196498] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4894.199241] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4894.207099] pending: sec_gc_main [ptlrpc] [ 4894.218808] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4894.237234] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4894.245680] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4894.250641] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4894.258382] workqueue events_long: flags=0x0 [ 4894.263383] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4894.271354] pending: lu_objects_flush [obdclass] [ 4894.274486] workqueue events_unbound: flags=0x2 [ 4894.277091] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4894.287813] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4894.297615] workqueue events_power_efficient: flags=0x80 [ 4894.302889] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4894.311062] pending: neigh_periodic_work, do_cache_clean [ 4894.313168] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4894.322074] pending: neigh_periodic_work, check_lifetime [ 4894.329167] workqueue mm_percpu_wq: flags=0x8 [ 4894.332355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4894.336146] pending: vmstat_update [ 4894.337905] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4894.346357] pending: vmstat_update [ 4894.353032] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4894.358800] pending: vmstat_update [ 4894.360711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4894.366141] pending: vmstat_update [ 4894.367653] workqueue writeback: flags=0x4a [ 4894.372121] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4894.382340] pending: wb_workfn, wb_workfn [ 4894.383949] workqueue xprtiod: flags=0xa [ 4894.385406] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4894.391027] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4894.400722] workqueue ipv6_addrconf: flags=0x40008 [ 4894.411494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4894.415309] pending: addrconf_verify_work [ 4894.427848] workqueue knbd0-recv: flags=0x1a [ 4894.429328] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4894.442096] in-flight: 647:recv_work [ 4894.443554] workqueue ptlrpc_pinger: flags=0x2 [ 4894.449136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4894.452990] pending: ptlrpc_pinger_main [ptlrpc] [ 4894.454892] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4884s workers=2 idle: 652 [ 4900.058410] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4900.068052] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4900.114742] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4900.120326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4900.125048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4900.134529] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4900.152080] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4900.161660] RAX: 000000000000730a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4900.165632] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4900.169461] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4900.186843] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4900.192761] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4900.203052] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4900.210825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4900.216259] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4900.219658] Call Trace: [ 4900.221997] [ 4900.225600] ? show_regs.cold.9+0x22/0x2f [ 4900.233275] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4900.234993] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4900.240176] ? watchdog+0x40/0x40 [ 4900.241366] ? hrtimer_interrupt+0x128/0x2c0 [ 4900.242886] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4900.251269] ? apic_timer_interrupt+0xf/0x20 [ 4900.254680] [ 4900.255785] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4900.259035] do_raw_spin_lock+0x177/0x190 [ 4900.264326] _raw_spin_lock+0x21/0x30 [ 4900.269015] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4900.276733] ? do_raw_spin_unlock+0x75/0x190 [ 4900.278168] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4900.292697] ? do_wait_intr+0xf0/0xf0 [ 4900.299730] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4900.307001] mdc_close+0x53b/0xd70 [mdc] [ 4900.312773] lmv_close+0x18d/0x490 [lmv] [ 4900.318623] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4900.322615] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4900.329946] ll_md_close+0x274/0x800 [lustre] [ 4900.337197] ll_file_release+0x128/0xc50 [lustre] [ 4900.340513] __fput+0x10d/0x440 [ 4900.341656] ____fput+0x12/0x20 [ 4900.344155] task_work_run+0xc6/0x110 [ 4900.347448] exit_to_usermode_loop+0x1dd/0x1f0 [ 4900.351044] do_syscall_64+0x3d6/0x440 [ 4900.352397] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4900.359867] RIP: 0033:0x7f1a63983cd8 [ 4900.365315] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4900.379798] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4900.386968] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4900.390549] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4900.395875] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4900.401910] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4900.407032] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4908.058009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 4908.064711] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4908.110904] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4908.124182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4908.140293] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4908.145298] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4908.160955] RSP: 0018:ffffc38542073af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4908.176607] RAX: 0000000000002bf8 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4908.179135] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4908.196662] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4908.199764] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4908.205834] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4908.208203] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4908.215125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4908.219149] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4908.222208] Call Trace: [ 4908.225658] [ 4908.230080] ? show_regs.cold.9+0x22/0x2f [ 4908.231469] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4908.233154] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4908.244264] ? watchdog+0x40/0x40 [ 4908.251918] ? hrtimer_interrupt+0x128/0x2c0 [ 4908.258703] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4908.270900] ? apic_timer_interrupt+0xf/0x20 [ 4908.274650] [ 4908.276787] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4908.278594] do_raw_spin_lock+0x177/0x190 [ 4908.288479] _raw_spin_lock+0x21/0x30 [ 4908.290825] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4908.293132] ? do_raw_spin_unlock+0x75/0x190 [ 4908.294996] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4908.301287] ? do_wait_intr+0xf0/0xf0 [ 4908.302954] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4908.305315] mdc_close+0x53b/0xd70 [mdc] [ 4908.307542] lmv_close+0x18d/0x490 [lmv] [ 4908.309260] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4908.311987] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4908.314120] ll_md_close+0x274/0x800 [lustre] [ 4908.320135] ll_file_release+0x128/0xc50 [lustre] [ 4908.322184] __fput+0x10d/0x440 [ 4908.323676] ____fput+0x12/0x20 [ 4908.325092] task_work_run+0xc6/0x110 [ 4908.326734] exit_to_usermode_loop+0x1dd/0x1f0 [ 4908.329873] do_syscall_64+0x3d6/0x440 [ 4908.331982] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4908.337616] RIP: 0033:0x7f1a63983cd8 [ 4908.338899] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4908.346807] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4908.349737] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4908.352435] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4908.362659] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4908.368750] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4908.371296] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4920.053023] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 4920.055947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4920.059256] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4920.091904] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4920.095969] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul [ 4920.106663] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4920.106673] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4920.106688] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4920.106692] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4920.106698] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4920.106701] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4920.106703] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4920.106706] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4920.106709] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4920.106712] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4920.106715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4920.106717] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4920.106725] Call Trace: [ 4920.106730] [ 4920.106736] ? show_regs.cold.9+0x22/0x2f [ 4920.106745] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4920.106750] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4920.106755] ? watchdog+0x40/0x40 [ 4920.106759] ? hrtimer_interrupt+0x128/0x2c0 [ 4920.106762] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4920.106769] ? apic_timer_interrupt+0xf/0x20 [ 4920.106773] [ 4920.106775] ? native_safe_halt+0x2e/0x50 [ 4920.106779] kvm_wait+0xcd/0xe0 [ 4920.106784] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4920.106792] do_raw_spin_lock+0x177/0x190 [ 4920.106796] _raw_spin_lock+0x21/0x30 [ 4920.106800] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4920.107064] ? do_raw_spin_unlock+0x75/0x190 [ 4920.107069] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4920.141552] crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4920.141581] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4920.141589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4920.141592] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4920.141605] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4920.141614] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4920.141618] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4920.141620] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4920.141622] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4920.141624] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4920.141627] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4920.141630] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4920.141633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4920.141635] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4920.141643] Call Trace: [ 4920.141648] [ 4920.141653] ? show_regs.cold.9+0x22/0x2f [ 4920.141664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4920.141671] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4920.141675] ? watchdog+0x40/0x40 [ 4920.141679] ? hrtimer_interrupt+0x128/0x2c0 [ 4920.141683] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4920.141692] ? apic_timer_interrupt+0xf/0x20 [ 4920.141697] [ 4920.141700] ? native_safe_halt+0x2e/0x50 [ 4920.141704] kvm_wait+0xcd/0xe0 [ 4920.141712] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4920.141720] do_raw_spin_lock+0x177/0x190 [ 4920.141725] _raw_spin_lock+0x21/0x30 [ 4920.141731] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4920.141998] obd_mod_free+0x46/0x140 [ptlrpc] [ 4920.142717] mdc_commit_open+0x91/0xb0 [mdc] [ 4920.142752] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4920.142954] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4920.146466] after_reply+0xe2c/0x1780 [ptlrpc] [ 4920.147673] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4920.147897] ? __next_timer_interrupt+0x160/0x160 [ 4920.147907] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4920.162824] ? do_wait_intr+0xf0/0xf0 [ 4920.162845] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4920.164424] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4920.167037] ? do_wait_intr+0xf0/0xf0 [ 4920.179116] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4920.180286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4920.180547] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4920.180734] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4920.184056] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4920.187879] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4920.206703] kthread+0x1d1/0x200 [ 4920.206722] ? set_kthread_struct+0x70/0x70 [ 4920.206727] ret_from_fork+0x1f/0x30 [ 4920.441472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4920.446301] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4920.452517] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4920.456734] lmv_intent_open+0x338/0x1150 [lmv] [ 4920.459165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4920.462028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4920.464856] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4920.468078] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4920.475427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4920.482510] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4920.490535] ? do_raw_spin_unlock+0x75/0x190 [ 4920.497381] ? _raw_spin_unlock+0x12/0x30 [ 4920.500917] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4920.503738] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4920.508937] ll_atomic_open+0x36a/0xd70 [lustre] [ 4920.510813] lookup_open+0x356/0xb40 [ 4920.512134] do_last+0x313/0xfc0 [ 4920.519653] ? nd_jump_root+0xe5/0x160 [ 4920.521109] ? path_init+0x437/0x520 [ 4920.524674] path_openat+0xf7/0x500 [ 4920.526717] ? do_raw_spin_unlock+0x75/0x190 [ 4920.529682] do_filp_open+0x99/0x140 [ 4920.533562] ? getname_flags+0x6e/0x330 [ 4920.544532] ? __check_object_size+0xff/0x256 [ 4920.546892] ? do_raw_spin_unlock+0x75/0x190 [ 4920.555567] ? _raw_spin_unlock+0x12/0x30 [ 4920.559991] do_sys_openat2+0x2b4/0x410 [ 4920.562883] do_sys_open+0x73/0xa0 [ 4920.564748] __x64_sys_openat+0x24/0x30 [ 4920.568481] do_syscall_64+0xc1/0x440 [ 4920.571824] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4920.576460] RIP: 0033:0x7f1a63983332 [ 4920.586966] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4920.603230] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4920.610455] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4920.616134] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4920.621384] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4920.628373] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4920.630973] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4924.897108] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2901s! [ 4924.902892] Showing busy workqueues and worker pools: [ 4924.905759] workqueue events: flags=0x0 [ 4924.908334] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4924.911555] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4924.914731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4924.919847] pending: sec_gc_main [ptlrpc] [ 4924.923194] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4924.930269] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4924.934445] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4924.940456] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4924.945295] workqueue events_long: flags=0x0 [ 4924.950592] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4924.959633] pending: lu_objects_flush [obdclass] [ 4924.963036] workqueue events_unbound: flags=0x2 [ 4924.966418] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4924.969092] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4924.974223] workqueue events_power_efficient: flags=0x80 [ 4924.976445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4924.979304] pending: neigh_periodic_work, do_cache_clean [ 4924.981491] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4924.984461] pending: neigh_periodic_work, check_lifetime [ 4924.986512] workqueue mm_percpu_wq: flags=0x8 [ 4924.987718] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4924.989823] pending: vmstat_update [ 4924.990971] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4924.993356] pending: vmstat_update [ 4924.994489] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4924.996429] pending: vmstat_update [ 4924.997450] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4924.999793] pending: vmstat_update [ 4925.001345] workqueue writeback: flags=0x4a [ 4925.003411] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4925.005937] pending: wb_workfn, wb_workfn [ 4925.007890] workqueue xprtiod: flags=0xa [ 4925.009468] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4925.012695] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4925.016087] workqueue ipv6_addrconf: flags=0x40008 [ 4925.017977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4925.020886] pending: addrconf_verify_work [ 4925.023422] workqueue knbd0-recv: flags=0x1a [ 4925.025353] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4925.028038] in-flight: 647:recv_work [ 4925.029881] workqueue ptlrpc_pinger: flags=0x2 [ 4925.031510] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4925.033577] pending: ptlrpc_pinger_main [ptlrpc] [ 4925.035701] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4915s workers=2 idle: 652 [ 4928.057488] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4928.063028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4928.083461] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4928.088867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4928.092169] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4928.094326] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4928.105090] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4928.108065] RAX: 0000000000006680 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4928.110391] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4928.112664] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4928.114633] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4928.116676] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4928.118700] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4928.120799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4928.122506] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4928.124539] Call Trace: [ 4928.125197] [ 4928.125753] ? show_regs.cold.9+0x22/0x2f [ 4928.126873] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4928.128865] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4928.130252] ? watchdog+0x40/0x40 [ 4928.131096] ? hrtimer_interrupt+0x128/0x2c0 [ 4928.132551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4928.133842] ? apic_timer_interrupt+0xf/0x20 [ 4928.135706] [ 4928.136311] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4928.137812] do_raw_spin_lock+0x177/0x190 [ 4928.138902] _raw_spin_lock+0x21/0x30 [ 4928.139848] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4928.141409] ? do_raw_spin_unlock+0x75/0x190 [ 4928.142587] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4928.144251] ? do_wait_intr+0xf0/0xf0 [ 4928.145252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4928.146769] mdc_close+0x53b/0xd70 [mdc] [ 4928.147799] lmv_close+0x18d/0x490 [lmv] [ 4928.148851] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4928.150438] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4928.151732] ll_md_close+0x274/0x800 [lustre] [ 4928.153489] ll_file_release+0x128/0xc50 [lustre] [ 4928.155390] __fput+0x10d/0x440 [ 4928.157092] ____fput+0x12/0x20 [ 4928.158225] task_work_run+0xc6/0x110 [ 4928.159749] exit_to_usermode_loop+0x1dd/0x1f0 [ 4928.161385] do_syscall_64+0x3d6/0x440 [ 4928.163004] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4928.165530] RIP: 0033:0x7f1a63983cd8 [ 4928.166883] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4928.172253] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4928.174646] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4928.178326] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4928.181234] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4928.185837] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4928.187977] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4936.058397] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4936.072592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4936.131938] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4936.148085] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4936.159329] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4936.167568] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4936.191123] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4936.198590] RAX: 0000000000000f68 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4936.203204] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4936.207524] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4936.214571] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4936.226087] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4936.236868] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4936.246565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4936.267821] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4936.282496] Call Trace: [ 4936.283838] [ 4936.286840] ? show_regs.cold.9+0x22/0x2f [ 4936.292256] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4936.295624] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4936.298632] ? watchdog+0x40/0x40 [ 4936.302314] ? hrtimer_interrupt+0x128/0x2c0 [ 4936.304339] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4936.314581] ? apic_timer_interrupt+0xf/0x20 [ 4936.317334] [ 4936.318364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4936.323778] do_raw_spin_lock+0x177/0x190 [ 4936.326898] _raw_spin_lock+0x21/0x30 [ 4936.329263] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4936.332883] ? do_raw_spin_unlock+0x75/0x190 [ 4936.339010] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4936.342511] ? do_wait_intr+0xf0/0xf0 [ 4936.347130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4936.351842] mdc_close+0x53b/0xd70 [mdc] [ 4936.354692] lmv_close+0x18d/0x490 [lmv] [ 4936.357576] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4936.361543] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4936.364581] ll_md_close+0x274/0x800 [lustre] [ 4936.370975] ll_file_release+0x128/0xc50 [lustre] [ 4936.376233] __fput+0x10d/0x440 [ 4936.380364] ____fput+0x12/0x20 [ 4936.385521] task_work_run+0xc6/0x110 [ 4936.390722] exit_to_usermode_loop+0x1dd/0x1f0 [ 4936.396911] do_syscall_64+0x3d6/0x440 [ 4936.399018] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4936.414509] RIP: 0033:0x7f1a63983cd8 [ 4936.421026] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4936.450752] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4936.459208] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4936.467810] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4936.480064] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4936.487542] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4936.497512] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4948.058191] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 4948.061733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4948.083079] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4948.087921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4948.092189] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4948.094317] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4948.103375] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4948.105648] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4948.113514] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4948.115912] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4948.121206] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4948.126713] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4948.130809] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4948.136165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4948.138137] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4948.143729] Call Trace: [ 4948.145312] [ 4948.145819] ? show_regs.cold.9+0x22/0x2f [ 4948.147332] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4948.148769] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4948.152570] ? watchdog+0x40/0x40 [ 4948.154257] ? hrtimer_interrupt+0x128/0x2c0 [ 4948.159284] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4948.162136] ? apic_timer_interrupt+0xf/0x20 [ 4948.163492] [ 4948.167694] ? native_safe_halt+0x2e/0x50 [ 4948.169634] kvm_wait+0xcd/0xe0 [ 4948.171150] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4948.174872] do_raw_spin_lock+0x177/0x190 [ 4948.177284] _raw_spin_lock+0x21/0x30 [ 4948.178793] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4948.181368] obd_mod_free+0x46/0x140 [ptlrpc] [ 4948.185201] mdc_commit_open+0x91/0xb0 [mdc] [ 4948.186964] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4948.189311] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4948.191245] after_reply+0xe2c/0x1780 [ptlrpc] [ 4948.195384] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4948.201328] ? __next_timer_interrupt+0x160/0x160 [ 4948.204660] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4948.207794] ? do_wait_intr+0xf0/0xf0 [ 4948.210008] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4948.211646] kthread+0x1d1/0x200 [ 4948.214533] ? set_kthread_struct+0x70/0x70 [ 4948.215787] ret_from_fork+0x1f/0x30 [ 4952.053011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 4952.058585] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4952.103223] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4952.115345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4952.122932] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4952.127514] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4952.147856] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4952.154238] RAX: 0000000000005106 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4952.159502] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4952.164269] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4952.168674] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4952.171722] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4952.176731] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4952.182984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4952.188886] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4952.195853] Call Trace: [ 4952.198240] [ 4952.200453] ? show_regs.cold.9+0x22/0x2f [ 4952.203313] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4952.206535] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4952.209488] ? watchdog+0x40/0x40 [ 4952.211707] ? hrtimer_interrupt+0x128/0x2c0 [ 4952.214517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4952.218548] ? apic_timer_interrupt+0xf/0x20 [ 4952.221372] [ 4952.223058] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4952.228658] do_raw_spin_lock+0x177/0x190 [ 4952.231880] _raw_spin_lock+0x21/0x30 [ 4952.235508] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4952.240685] ? do_raw_spin_unlock+0x75/0x190 [ 4952.245539] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4952.251567] ? do_wait_intr+0xf0/0xf0 [ 4952.253676] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4952.258604] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4952.264437] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4952.269557] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4952.274533] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4952.279520] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4952.283220] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4952.286421] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4952.291568] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4952.294513] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4952.298992] lmv_intent_open+0x338/0x1150 [lmv] [ 4952.302877] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4952.305623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4952.309611] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4952.312599] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4952.316229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4952.319875] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4952.322664] ? do_raw_spin_unlock+0x75/0x190 [ 4952.329397] ? _raw_spin_unlock+0x12/0x30 [ 4952.330956] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4952.333741] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4952.336604] ll_atomic_open+0x36a/0xd70 [lustre] [ 4952.339633] lookup_open+0x356/0xb40 [ 4952.341598] do_last+0x313/0xfc0 [ 4952.343513] ? nd_jump_root+0xe5/0x160 [ 4952.345992] ? path_init+0x437/0x520 [ 4952.348609] path_openat+0xf7/0x500 [ 4952.351313] ? do_raw_spin_unlock+0x75/0x190 [ 4952.353015] do_filp_open+0x99/0x140 [ 4952.354325] ? getname_flags+0x6e/0x330 [ 4952.356736] ? __check_object_size+0xff/0x256 [ 4952.359680] ? do_raw_spin_unlock+0x75/0x190 [ 4952.362727] ? _raw_spin_unlock+0x12/0x30 [ 4952.365275] do_sys_openat2+0x2b4/0x410 [ 4952.367499] do_sys_open+0x73/0xa0 [ 4952.369421] __x64_sys_openat+0x24/0x30 [ 4952.371446] do_syscall_64+0xc1/0x440 [ 4952.373574] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4952.376529] RIP: 0033:0x7f1a63983332 [ 4952.379489] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4952.388557] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4952.393428] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4952.398242] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4952.403284] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4952.407951] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4952.412256] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4955.105156] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2932s! [ 4955.109628] Showing busy workqueues and worker pools: [ 4955.114326] workqueue events: flags=0x0 [ 4955.116956] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4955.120434] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4955.124348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4955.127437] pending: sec_gc_main [ptlrpc] [ 4955.130576] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4955.134157] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4955.139412] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4955.142535] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4955.149011] workqueue events_long: flags=0x0 [ 4955.151567] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4955.159306] pending: lu_objects_flush [obdclass] [ 4955.168427] workqueue events_unbound: flags=0x2 [ 4955.171320] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4955.178292] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4955.188329] workqueue events_power_efficient: flags=0x80 [ 4955.190689] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4955.194540] pending: neigh_periodic_work, do_cache_clean [ 4955.198298] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4955.201631] pending: neigh_periodic_work, check_lifetime [ 4955.204483] workqueue mm_percpu_wq: flags=0x8 [ 4955.206646] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4955.210409] pending: vmstat_update [ 4955.212444] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4955.217056] pending: vmstat_update [ 4955.218592] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4955.221559] pending: vmstat_update [ 4955.223268] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4955.226193] pending: vmstat_update [ 4955.228369] workqueue writeback: flags=0x4a [ 4955.236086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4955.253827] pending: wb_workfn, wb_workfn [ 4955.257201] workqueue xprtiod: flags=0xa [ 4955.259649] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4955.272022] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4955.281276] workqueue ipv6_addrconf: flags=0x40008 [ 4955.284196] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4955.287465] pending: addrconf_verify_work [ 4955.289757] workqueue knbd0-recv: flags=0x1a [ 4955.293183] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4955.298475] in-flight: 647:recv_work [ 4955.302298] workqueue ptlrpc_pinger: flags=0x2 [ 4955.305975] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4955.308420] pending: ptlrpc_pinger_main [ptlrpc] [ 4955.313534] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4945s workers=2 idle: 652 [ 4956.057482] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4956.060155] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4956.084308] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4956.088345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4956.091449] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4956.092975] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4956.099738] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4956.102068] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4956.105298] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4956.108409] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4956.110636] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4956.114437] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4956.123590] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4956.135756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4956.140687] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4956.152152] Call Trace: [ 4956.156308] [ 4956.159093] ? show_regs.cold.9+0x22/0x2f [ 4956.161155] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4956.168066] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4956.170421] ? watchdog+0x40/0x40 [ 4956.172820] ? hrtimer_interrupt+0x128/0x2c0 [ 4956.175668] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4956.177726] ? apic_timer_interrupt+0xf/0x20 [ 4956.184228] [ 4956.185197] ? native_safe_halt+0x2e/0x50 [ 4956.188320] kvm_wait+0xcd/0xe0 [ 4956.192445] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4956.198809] do_raw_spin_lock+0x177/0x190 [ 4956.202809] _raw_spin_lock+0x21/0x30 [ 4956.207120] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4956.209991] ? do_raw_spin_unlock+0x75/0x190 [ 4956.211664] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4956.218059] ? do_wait_intr+0xf0/0xf0 [ 4956.220076] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4956.223758] mdc_close+0x53b/0xd70 [mdc] [ 4956.226301] lmv_close+0x18d/0x490 [lmv] [ 4956.229392] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4956.232734] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4956.235765] ll_md_close+0x274/0x800 [lustre] [ 4956.239110] ll_file_release+0x128/0xc50 [lustre] [ 4956.244616] __fput+0x10d/0x440 [ 4956.246731] ____fput+0x12/0x20 [ 4956.248683] task_work_run+0xc6/0x110 [ 4956.252845] exit_to_usermode_loop+0x1dd/0x1f0 [ 4956.255394] do_syscall_64+0x3d6/0x440 [ 4956.257188] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4956.259723] RIP: 0033:0x7f1a63983cd8 [ 4956.261243] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4956.270448] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4956.274366] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4956.278156] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4956.282725] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4956.284410] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4956.292269] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4962.476179] rcu: INFO: rcu_sched self-detected stall on CPU [ 4962.481677] rcu: 3-...!: (1770711 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 4962.486605] (t=2940077 jiffies g=269633 q=1199) [ 4962.489301] rcu: rcu_sched kthread starved for 2939729 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 4962.500110] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4962.506279] rcu: RCU grace-period kthread stack dump: [ 4962.508792] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4962.514337] Call Trace: [ 4962.515338] __schedule+0x351/0xcb0 [ 4962.516599] schedule+0xc0/0x180 [ 4962.518199] schedule_timeout+0xb4/0x190 [ 4962.519706] ? do_raw_spin_unlock+0x75/0x190 [ 4962.521555] ? __next_timer_interrupt+0x160/0x160 [ 4962.523283] ? prepare_to_swait_event+0xf3/0x140 [ 4962.525120] rcu_gp_kthread+0x8dd/0x15b0 [ 4962.526562] ? force_qs_rnp+0x2c0/0x2c0 [ 4962.528637] kthread+0x1d1/0x200 [ 4962.530065] ? set_kthread_struct+0x70/0x70 [ 4962.532108] ret_from_fork+0x1f/0x30 [ 4962.533690] rcu: Stack dump where RCU GP kthread last ran: [ 4962.535996] Sending NMI from CPU 3 to CPUs 0: [ 4962.540001] NMI backtrace for cpu 0 [ 4962.540007] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4962.540009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4962.540010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4962.540012] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4962.540014] RSP: 0018:ffffc38541c37500 EFLAGS: 00000216 [ 4962.540017] RAX: 0000000000007a97 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4962.540018] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 4962.540020] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 4962.540021] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 4962.540023] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 4962.540024] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 4962.540026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4962.540027] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 4962.540028] Call Trace: [ 4962.540029] [ 4962.540030] ? show_regs.cold.9+0x22/0x2f [ 4962.540031] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4962.540033] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4962.540033] ? nmi_handle+0xbd/0x210 [ 4962.540035] ? default_do_nmi+0x4b/0x1b0 [ 4962.540036] ? do_nmi+0x280/0x380 [ 4962.540037] ? end_repeat_nmi+0x16/0x1f [ 4962.540038] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4962.540039] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4962.540040] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4962.540042] [ 4962.540043] do_raw_spin_lock+0x177/0x190 [ 4962.540044] _raw_spin_lock+0x21/0x30 [ 4962.540045] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4962.540046] ? do_raw_spin_unlock+0x75/0x190 [ 4962.540047] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4962.540048] ? do_wait_intr+0xf0/0xf0 [ 4962.540049] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4962.540050] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4962.540051] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4962.540052] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4962.540053] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4962.540055] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4962.540056] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4962.540057] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4962.540059] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4962.540060] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4962.540061] lmv_intent_open+0x338/0x1150 [lmv] [ 4962.540062] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4962.540063] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4962.540064] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4962.540066] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4962.540067] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4962.540068] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4962.540069] ? do_raw_spin_unlock+0x75/0x190 [ 4962.540070] ? _raw_spin_unlock+0x12/0x30 [ 4962.540072] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4962.540073] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4962.540074] ll_atomic_open+0x36a/0xd70 [lustre] [ 4962.540075] lookup_open+0x356/0xb40 [ 4962.540076] do_last+0x313/0xfc0 [ 4962.540077] ? nd_jump_root+0xe5/0x160 [ 4962.540078] ? path_init+0x437/0x520 [ 4962.540079] path_openat+0xf7/0x500 [ 4962.540080] ? do_raw_spin_unlock+0x75/0x190 [ 4962.540081] do_filp_open+0x99/0x140 [ 4962.540082] ? getname_flags+0x6e/0x330 [ 4962.540083] ? __check_object_size+0xff/0x256 [ 4962.540084] ? do_raw_spin_unlock+0x75/0x190 [ 4962.540085] ? _raw_spin_unlock+0x12/0x30 [ 4962.540086] do_sys_openat2+0x2b4/0x410 [ 4962.540088] do_sys_open+0x73/0xa0 [ 4962.540089] __x64_sys_openat+0x24/0x30 [ 4962.540089] do_syscall_64+0xc1/0x440 [ 4962.540090] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4962.540092] RIP: 0033:0x7f1a63983332 [ 4962.540093] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4962.540095] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4962.540098] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 4962.540099] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 4962.540101] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 4962.540102] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 4962.540104] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 4962.540727] NMI backtrace for cpu 3 [ 4962.910559] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4962.915710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4962.922713] Call Trace: [ 4962.924005] [ 4962.925994] ? dump_stack+0xbb/0x10e [ 4962.931750] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4962.935391] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4962.939935] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4962.941433] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4962.949273] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4962.953064] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4962.955462] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4962.961066] ? update_process_times+0x6d/0xc0 [ 4962.964735] ? tick_sched_handle.isra.18+0x27/0x90 [ 4962.967666] ? tick_sched_timer+0x53/0xb0 [ 4962.968661] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4962.969870] ? tick_sched_handle.isra.18+0x90/0x90 [ 4962.976558] ? hrtimer_interrupt+0x128/0x2c0 [ 4962.980423] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4962.984539] ? apic_timer_interrupt+0xf/0x20 [ 4962.987551] [ 4962.989494] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4962.991701] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4962.993759] ? do_raw_spin_lock+0x177/0x190 [ 4962.996092] ? _raw_spin_lock+0x21/0x30 [ 4962.998251] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4963.001757] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4963.009230] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4963.011533] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4963.013741] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4963.018525] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4963.020929] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4963.024005] ? __next_timer_interrupt+0x160/0x160 [ 4963.026412] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4963.029056] ? do_wait_intr+0xf0/0xf0 [ 4963.030977] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4963.033286] ? kthread+0x1d1/0x200 [ 4963.034624] ? set_kthread_struct+0x70/0x70 [ 4963.038087] ? ret_from_fork+0x1f/0x30 [ 4964.056399] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4964.062718] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4964.137375] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4964.146060] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4964.151931] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4964.157765] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4964.166059] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4964.170225] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4964.173195] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4964.176601] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4964.180432] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4964.183635] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4964.187184] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4964.192740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4964.195201] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4964.199027] Call Trace: [ 4964.200256] [ 4964.201721] ? show_regs.cold.9+0x22/0x2f [ 4964.206632] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4964.210518] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4964.216200] ? watchdog+0x40/0x40 [ 4964.220566] ? hrtimer_interrupt+0x128/0x2c0 [ 4964.223657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4964.230921] ? apic_timer_interrupt+0xf/0x20 [ 4964.236641] [ 4964.238933] ? native_safe_halt+0x2e/0x50 [ 4964.245974] kvm_wait+0xcd/0xe0 [ 4964.248023] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4964.255264] do_raw_spin_lock+0x177/0x190 [ 4964.260438] _raw_spin_lock+0x21/0x30 [ 4964.263315] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4964.267723] ? do_raw_spin_unlock+0x75/0x190 [ 4964.271372] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4964.278762] ? do_wait_intr+0xf0/0xf0 [ 4964.280597] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4964.289936] mdc_close+0x53b/0xd70 [mdc] [ 4964.293625] lmv_close+0x18d/0x490 [lmv] [ 4964.297862] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4964.301729] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4964.303700] ll_md_close+0x274/0x800 [lustre] [ 4964.309332] ll_file_release+0x128/0xc50 [lustre] [ 4964.313804] __fput+0x10d/0x440 [ 4964.315342] ____fput+0x12/0x20 [ 4964.316846] task_work_run+0xc6/0x110 [ 4964.318840] exit_to_usermode_loop+0x1dd/0x1f0 [ 4964.321192] do_syscall_64+0x3d6/0x440 [ 4964.322750] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4964.325192] RIP: 0033:0x7f1a63983cd8 [ 4964.326720] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4964.336739] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4964.340497] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4964.343484] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4964.346729] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4964.350581] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4964.354658] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4984.057035] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 4984.060505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4984.103905] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4984.111962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4984.120708] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4984.127131] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 4984.138387] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4984.143073] RAX: 0000000000005cb7 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4984.148655] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 4984.154311] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 4984.159706] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 4984.170208] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 4984.177623] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 4984.184664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4984.191175] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 4984.203407] Call Trace: [ 4984.206757] [ 4984.210005] ? show_regs.cold.9+0x22/0x2f [ 4984.215866] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4984.227952] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4984.230779] ? watchdog+0x40/0x40 [ 4984.233967] ? hrtimer_interrupt+0x128/0x2c0 [ 4984.244050] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4984.250512] ? apic_timer_interrupt+0xf/0x20 [ 4984.251998] [ 4984.252599] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4984.259339] do_raw_spin_lock+0x177/0x190 [ 4984.261819] _raw_spin_lock+0x21/0x30 [ 4984.263558] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4984.269648] ? do_raw_spin_unlock+0x75/0x190 [ 4984.276394] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4984.279658] ? do_wait_intr+0xf0/0xf0 [ 4984.287055] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4984.289822] mdc_close+0x53b/0xd70 [mdc] [ 4984.292748] lmv_close+0x18d/0x490 [lmv] [ 4984.294458] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4984.298555] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4984.303040] ll_md_close+0x274/0x800 [lustre] [ 4984.306463] ll_file_release+0x128/0xc50 [lustre] [ 4984.308900] __fput+0x10d/0x440 [ 4984.311353] ____fput+0x12/0x20 [ 4984.314495] task_work_run+0xc6/0x110 [ 4984.317943] exit_to_usermode_loop+0x1dd/0x1f0 [ 4984.322150] do_syscall_64+0x3d6/0x440 [ 4984.324028] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4984.328040] RIP: 0033:0x7f1a63983cd8 [ 4984.330750] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4984.343111] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4984.348116] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 4984.353969] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4984.363100] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 4984.367051] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 4984.372935] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4985.828067] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2962s! [ 4985.832053] Showing busy workqueues and worker pools: [ 4985.834676] workqueue events: flags=0x0 [ 4985.836319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4985.838608] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 4985.841691] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4985.845229] pending: sec_gc_main [ptlrpc] [ 4985.847680] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4985.851225] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 4985.856375] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4985.860221] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 4985.865042] workqueue events_long: flags=0x0 [ 4985.867320] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4985.870943] pending: lu_objects_flush [obdclass] [ 4985.873869] workqueue events_unbound: flags=0x2 [ 4985.875700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4985.879756] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4985.887579] workqueue events_power_efficient: flags=0x80 [ 4985.890785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4985.894980] pending: neigh_periodic_work, do_cache_clean [ 4985.898183] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4985.901220] pending: neigh_periodic_work, check_lifetime [ 4985.904205] workqueue mm_percpu_wq: flags=0x8 [ 4985.906501] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4985.910476] pending: vmstat_update [ 4985.912787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4985.916454] pending: vmstat_update [ 4985.918457] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4985.921892] pending: vmstat_update [ 4985.923756] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4985.927230] pending: vmstat_update [ 4985.929257] workqueue writeback: flags=0x4a [ 4985.931250] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4985.934088] pending: wb_workfn, wb_workfn [ 4985.936054] workqueue xprtiod: flags=0xa [ 4985.937932] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4985.940753] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 4985.945178] workqueue ipv6_addrconf: flags=0x40008 [ 4985.949034] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4985.951357] pending: addrconf_verify_work [ 4985.952903] workqueue knbd0-recv: flags=0x1a [ 4985.955097] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4985.962186] in-flight: 647:recv_work [ 4985.965036] workqueue ptlrpc_pinger: flags=0x2 [ 4985.968041] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4985.972036] pending: ptlrpc_pinger_main [ptlrpc] [ 4985.975021] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4976s workers=2 idle: 652 [ 4988.058212] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 4988.061068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4988.081430] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4988.086232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4988.090189] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4988.092428] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4988.099860] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4988.103154] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4988.106052] RDX: 0000000000007a39 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 4988.109177] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 4988.111787] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 4988.114509] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4988.117082] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 4988.119861] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4988.121814] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 4988.124713] Call Trace: [ 4988.125787] [ 4988.127033] ? show_regs.cold.9+0x22/0x2f [ 4988.128764] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4988.130851] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4988.132780] ? watchdog+0x40/0x40 [ 4988.134122] ? hrtimer_interrupt+0x128/0x2c0 [ 4988.135874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4988.137807] ? apic_timer_interrupt+0xf/0x20 [ 4988.139494] [ 4988.140202] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4988.142957] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4988.147711] do_raw_spin_lock+0x177/0x190 [ 4988.149415] _raw_spin_lock+0x21/0x30 [ 4988.163820] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4988.166874] obd_mod_free+0x46/0x140 [ptlrpc] [ 4988.169778] mdc_commit_open+0x91/0xb0 [mdc] [ 4988.174262] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4988.178412] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4988.182390] after_reply+0xe2c/0x1780 [ptlrpc] [ 4988.185005] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4988.187183] ? __next_timer_interrupt+0x160/0x160 [ 4988.189060] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4988.197843] ? do_wait_intr+0xf0/0xf0 [ 4988.203706] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4988.208092] kthread+0x1d1/0x200 [ 4988.212594] ? set_kthread_struct+0x70/0x70 [ 4988.215208] ret_from_fork+0x1f/0x30 [ 4992.056011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 4992.059858] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4992.108364] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4992.119936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4992.126358] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4992.129688] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4992.137377] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4992.150487] RAX: 000000000000770b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 4992.156022] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 4992.166660] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4992.171307] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 4992.180728] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 4992.184094] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 4992.192006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4992.202982] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 4992.210003] Call Trace: [ 4992.211084] [ 4992.211899] ? show_regs.cold.9+0x22/0x2f [ 4992.221737] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4992.229310] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4992.234952] ? watchdog+0x40/0x40 [ 4992.238185] ? hrtimer_interrupt+0x128/0x2c0 [ 4992.243108] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4992.245798] ? apic_timer_interrupt+0xf/0x20 [ 4992.250239] [ 4992.253123] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4992.255201] do_raw_spin_lock+0x177/0x190 [ 4992.258618] _raw_spin_lock+0x21/0x30 [ 4992.262926] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4992.265750] ? do_raw_spin_unlock+0x75/0x190 [ 4992.269106] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4992.271885] ? do_wait_intr+0xf0/0xf0 [ 4992.273416] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4992.281070] mdc_close+0x53b/0xd70 [mdc] [ 4992.283114] lmv_close+0x18d/0x490 [lmv] [ 4992.284437] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4992.289182] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4992.291609] ll_md_close+0x274/0x800 [lustre] [ 4992.292927] ll_file_release+0x128/0xc50 [lustre] [ 4992.294485] __fput+0x10d/0x440 [ 4992.304385] ____fput+0x12/0x20 [ 4992.307365] task_work_run+0xc6/0x110 [ 4992.315594] exit_to_usermode_loop+0x1dd/0x1f0 [ 4992.316854] do_syscall_64+0x3d6/0x440 [ 4992.319188] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4992.322986] RIP: 0033:0x7f1a63983cd8 [ 4992.324142] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4992.332219] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4992.334948] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 4992.337418] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 4992.339760] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 4992.342621] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 4992.345080] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5012.053012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5012.055962] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5012.057010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5012.082106] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5012.086601] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5012.086693] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5012.086700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5012.086703] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5012.086718] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5012.086722] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5012.086727] RAX: 0000000000005d12 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5012.086730] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5012.086732] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5012.086740] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5012.086743] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5012.086746] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5012.086749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5012.086752] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5012.086760] Call Trace: [ 5012.086765] [ 5012.086771] ? show_regs.cold.9+0x22/0x2f [ 5012.086781] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5012.086787] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5012.086791] ? watchdog+0x40/0x40 [ 5012.086795] ? hrtimer_interrupt+0x128/0x2c0 [ 5012.086799] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5012.086807] ? apic_timer_interrupt+0xf/0x20 [ 5012.086812] [ 5012.086815] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5012.086820] do_raw_spin_lock+0x177/0x190 [ 5012.086824] _raw_spin_lock+0x21/0x30 [ 5012.086834] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5012.098181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5012.153219] ? do_raw_spin_unlock+0x75/0x190 [ 5012.153239] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5012.166755] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5012.166778] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5012.166782] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5012.166789] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5012.166791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5012.166793] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5012.166795] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5012.166797] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5012.166800] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5012.166803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5012.166806] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5012.166814] Call Trace: [ 5012.166820] [ 5012.166826] ? show_regs.cold.9+0x22/0x2f [ 5012.166836] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5012.166842] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5012.166847] ? watchdog+0x40/0x40 [ 5012.166851] ? hrtimer_interrupt+0x128/0x2c0 [ 5012.166855] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5012.166862] ? apic_timer_interrupt+0xf/0x20 [ 5012.166866] [ 5012.166869] ? native_safe_halt+0x2e/0x50 [ 5012.166873] kvm_wait+0xcd/0xe0 [ 5012.166880] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5012.166887] do_raw_spin_lock+0x177/0x190 [ 5012.166892] _raw_spin_lock+0x21/0x30 [ 5012.166896] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5012.167166] ? do_raw_spin_unlock+0x75/0x190 [ 5012.167172] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5012.167814] ? do_wait_intr+0xf0/0xf0 [ 5012.167823] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5012.168023] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5012.168240] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5012.168824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5012.168933] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5012.168970] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5012.169001] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5012.169029] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5012.169370] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5012.169834] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5012.169865] lmv_intent_open+0x338/0x1150 [lmv] [ 5012.169893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5012.169983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5012.170069] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5012.170093] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5012.170180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5012.170263] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5012.170783] ? do_raw_spin_unlock+0x75/0x190 [ 5012.170792] ? _raw_spin_unlock+0x12/0x30 [ 5012.170798] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5012.170829] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5012.170850] ll_atomic_open+0x36a/0xd70 [lustre] [ 5012.170934] lookup_open+0x356/0xb40 [ 5012.170944] do_last+0x313/0xfc0 [ 5012.170950] ? nd_jump_root+0xe5/0x160 [ 5012.170954] ? path_init+0x437/0x520 [ 5012.170958] path_openat+0xf7/0x500 [ 5012.170963] ? do_raw_spin_unlock+0x75/0x190 [ 5012.170967] do_filp_open+0x99/0x140 [ 5012.170972] ? getname_flags+0x6e/0x330 [ 5012.170977] ? __check_object_size+0xff/0x256 [ 5012.170984] ? do_raw_spin_unlock+0x75/0x190 [ 5012.170988] ? _raw_spin_unlock+0x12/0x30 [ 5012.170993] do_sys_openat2+0x2b4/0x410 [ 5012.170997] do_sys_open+0x73/0xa0 [ 5012.171001] __x64_sys_openat+0x24/0x30 [ 5012.171005] do_syscall_64+0xc1/0x440 [ 5012.171011] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5012.171016] RIP: 0033:0x7f1a63983332 [ 5012.171070] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5012.171076] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5012.171081] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5012.171083] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5012.171085] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5012.171086] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5012.171088] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5012.443662] ? do_wait_intr+0xf0/0xf0 [ 5012.444885] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5012.446657] mdc_close+0x53b/0xd70 [mdc] [ 5012.447904] lmv_close+0x18d/0x490 [lmv] [ 5012.449095] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5012.450766] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5012.452461] ll_md_close+0x274/0x800 [lustre] [ 5012.453800] ll_file_release+0x128/0xc50 [lustre] [ 5012.455133] __fput+0x10d/0x440 [ 5012.456022] ____fput+0x12/0x20 [ 5012.458020] task_work_run+0xc6/0x110 [ 5012.459143] exit_to_usermode_loop+0x1dd/0x1f0 [ 5012.460451] do_syscall_64+0x3d6/0x440 [ 5012.461573] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5012.463118] RIP: 0033:0x7f1a63983cd8 [ 5012.464152] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5012.470667] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5012.473264] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5012.475548] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5012.477815] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5012.479939] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5012.482278] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5016.038489] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2993s! [ 5016.058028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5016.058100] Showing busy workqueues and worker pools: [ 5016.069834] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5016.069913] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5016.069921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5016.069924] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5016.069938] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5016.069943] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5016.069947] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5016.069950] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5016.069952] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5016.069954] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5016.069957] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5016.069960] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5016.069963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5016.069965] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5016.069974] Call Trace: [ 5016.069979] [ 5016.069985] ? show_regs.cold.9+0x22/0x2f [ 5016.069995] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5016.070002] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5016.070107] ? watchdog+0x40/0x40 [ 5016.070115] ? hrtimer_interrupt+0x128/0x2c0 [ 5016.070120] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5016.070127] ? apic_timer_interrupt+0xf/0x20 [ 5016.070131] [ 5016.070135] ? native_safe_halt+0x2e/0x50 [ 5016.070139] kvm_wait+0xcd/0xe0 [ 5016.070147] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5016.070154] do_raw_spin_lock+0x177/0x190 [ 5016.070159] _raw_spin_lock+0x21/0x30 [ 5016.070164] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5016.080393] workqueue events: flags=0x0 [ 5016.080403] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5016.080417] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5016.080694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5016.080704] pending: sec_gc_main [ptlrpc] [ 5016.080909] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5016.080918] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5016.081302] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5016.081313] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5016.081578] workqueue events_long: flags=0x0 [ 5016.081582] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5016.081589] pending: lu_objects_flush [obdclass] [ 5016.081763] workqueue events_unbound: flags=0x2 [ 5016.081765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5016.081772] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5016.081793] workqueue events_power_efficient: flags=0x80 [ 5016.081796] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5016.081802] pending: neigh_periodic_work, do_cache_clean [ 5016.081815] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5016.081820] pending: neigh_periodic_work, check_lifetime [ 5016.081833] workqueue mm_percpu_wq: flags=0x8 [ 5016.081835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5016.081841] pending: vmstat_update [ 5016.081845] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5016.081851] pending: vmstat_update [ 5016.081855] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5016.081861] pending: vmstat_update [ 5016.081865] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5016.081871] pending: vmstat_update [ 5016.081878] workqueue writeback: flags=0x4a [ 5016.081881] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5016.081886] pending: wb_workfn, wb_workfn [ 5016.081901] workqueue xprtiod: flags=0xa [ 5016.081903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5016.081907] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5016.081925] workqueue ipv6_addrconf: flags=0x40008 [ 5016.081927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5016.081932] pending: addrconf_verify_work [ 5016.081944] workqueue knbd0-recv: flags=0x1a [ 5016.081946] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5016.081951] in-flight: 647:recv_work [ 5016.081961] workqueue ptlrpc_pinger: flags=0x2 [ 5016.081963] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5016.081968] pending: ptlrpc_pinger_main [ptlrpc] [ 5016.082193] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5006s workers=2 idle: 652 [ 5016.489924] obd_mod_free+0x46/0x140 [ptlrpc] [ 5016.495584] mdc_commit_open+0x91/0xb0 [mdc] [ 5016.500045] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5016.504468] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5016.507150] after_reply+0xe2c/0x1780 [ptlrpc] [ 5016.510490] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5016.512858] ? __next_timer_interrupt+0x160/0x160 [ 5016.515099] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5016.522147] ? do_wait_intr+0xf0/0xf0 [ 5016.527392] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5016.533230] kthread+0x1d1/0x200 [ 5016.534921] ? set_kthread_struct+0x70/0x70 [ 5016.536939] ret_from_fork+0x1f/0x30 [ 5020.056030] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 5020.060170] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5020.086710] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5020.093553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5020.098664] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5020.102535] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5020.114263] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5020.118371] RAX: 0000000000005c86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5020.122957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5020.131039] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5020.137742] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5020.140675] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5020.144408] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5020.148423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5020.150688] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5020.157963] Call Trace: [ 5020.159417] [ 5020.161279] ? show_regs.cold.9+0x22/0x2f [ 5020.165373] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5020.167449] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5020.171140] ? watchdog+0x40/0x40 [ 5020.172802] ? hrtimer_interrupt+0x128/0x2c0 [ 5020.174849] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5020.176684] ? apic_timer_interrupt+0xf/0x20 [ 5020.179201] [ 5020.184567] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5020.186815] do_raw_spin_lock+0x177/0x190 [ 5020.188891] _raw_spin_lock+0x21/0x30 [ 5020.192212] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5020.201000] ? do_raw_spin_unlock+0x75/0x190 [ 5020.203063] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5020.206384] ? do_wait_intr+0xf0/0xf0 [ 5020.209411] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5020.212639] mdc_close+0x53b/0xd70 [mdc] [ 5020.215069] lmv_close+0x18d/0x490 [lmv] [ 5020.217610] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5020.222027] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5020.224870] ll_md_close+0x274/0x800 [lustre] [ 5020.227572] ll_file_release+0x128/0xc50 [lustre] [ 5020.231848] __fput+0x10d/0x440 [ 5020.233415] ____fput+0x12/0x20 [ 5020.235569] task_work_run+0xc6/0x110 [ 5020.237902] exit_to_usermode_loop+0x1dd/0x1f0 [ 5020.240345] do_syscall_64+0x3d6/0x440 [ 5020.242229] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5020.246013] RIP: 0033:0x7f1a63983cd8 [ 5020.248435] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5020.263318] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5020.267132] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5020.270809] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5020.274447] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5020.276956] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5020.280675] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5040.057035] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5040.060603] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5040.084833] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5040.095691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5040.100731] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5040.106688] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5040.115616] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5040.123174] RAX: 00000000000072bc RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5040.126553] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5040.130857] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5040.133465] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5040.137540] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5040.140726] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5040.144758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5040.147410] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5040.152298] Call Trace: [ 5040.153531] [ 5040.154300] ? show_regs.cold.9+0x22/0x2f [ 5040.155781] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5040.160425] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5040.163567] ? watchdog+0x40/0x40 [ 5040.166907] ? hrtimer_interrupt+0x128/0x2c0 [ 5040.169571] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5040.173378] ? apic_timer_interrupt+0xf/0x20 [ 5040.176178] [ 5040.176972] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5040.181341] do_raw_spin_lock+0x177/0x190 [ 5040.185892] _raw_spin_lock+0x21/0x30 [ 5040.188026] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5040.190873] ? do_raw_spin_unlock+0x75/0x190 [ 5040.194726] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5040.205299] ? do_wait_intr+0xf0/0xf0 [ 5040.206689] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5040.211916] mdc_close+0x53b/0xd70 [mdc] [ 5040.215749] lmv_close+0x18d/0x490 [lmv] [ 5040.217293] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5040.223495] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5040.227575] ll_md_close+0x274/0x800 [lustre] [ 5040.229294] ll_file_release+0x128/0xc50 [lustre] [ 5040.236030] __fput+0x10d/0x440 [ 5040.237732] ____fput+0x12/0x20 [ 5040.240994] task_work_run+0xc6/0x110 [ 5040.248791] exit_to_usermode_loop+0x1dd/0x1f0 [ 5040.253137] do_syscall_64+0x3d6/0x440 [ 5040.254521] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5040.269586] RIP: 0033:0x7f1a63983cd8 [ 5040.271714] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5040.288874] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5040.296291] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5040.298961] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5040.305770] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5040.315162] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5040.328817] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5044.055678] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5044.058036] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5044.060151] Modules linked in: lustre(O) [ 5044.065820] Modules linked in: [ 5044.065824] osp(O) ofd(O) [ 5044.067800] lustre(O) [ 5044.069858] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5044.069931] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5044.069939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5044.069954] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5044.069968] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5044.069972] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5044.069977] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5044.069979] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5044.069982] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5044.069990] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5044.069994] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5044.070011] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5044.070014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5044.070017] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5044.070025] Call Trace: [ 5044.070031] [ 5044.070042] ? show_regs.cold.9+0x22/0x2f [ 5044.070053] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5044.070061] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5044.070065] ? watchdog+0x40/0x40 [ 5044.070070] ? hrtimer_interrupt+0x128/0x2c0 [ 5044.070074] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5044.070083] ? apic_timer_interrupt+0xf/0x20 [ 5044.070088] [ 5044.070091] ? native_safe_halt+0x2e/0x50 [ 5044.070095] kvm_wait+0xcd/0xe0 [ 5044.070103] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5044.070111] do_raw_spin_lock+0x177/0x190 [ 5044.070116] _raw_spin_lock+0x21/0x30 [ 5044.070120] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5044.070584] ? do_raw_spin_unlock+0x75/0x190 [ 5044.070590] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5044.070798] ? do_wait_intr+0xf0/0xf0 [ 5044.070812] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5044.071229] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5044.071471] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5044.071768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5044.072102] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5044.072147] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5044.072174] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5044.072202] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5044.072284] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5044.074735] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5044.074815] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5044.074823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5044.074826] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5044.074844] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5044.074848] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5044.074853] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5044.074856] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5044.074858] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5044.074860] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5044.074862] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5044.074865] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5044.074868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5044.074870] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5044.074879] Call Trace: [ 5044.074884] [ 5044.074889] ? show_regs.cold.9+0x22/0x2f [ 5044.074900] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5044.074906] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5044.074911] ? watchdog+0x40/0x40 [ 5044.074915] ? hrtimer_interrupt+0x128/0x2c0 [ 5044.074920] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5044.074929] ? apic_timer_interrupt+0xf/0x20 [ 5044.074933] [ 5044.074937] ? native_safe_halt+0x2e/0x50 [ 5044.074952] kvm_wait+0xcd/0xe0 [ 5044.109429] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5044.109482] lmv_intent_open+0x338/0x1150 [lmv] [ 5044.109528] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5044.109643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5044.109728] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5044.109751] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5044.109838] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5044.110074] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5044.118167] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5044.118193] do_raw_spin_lock+0x177/0x190 [ 5044.118199] _raw_spin_lock+0x21/0x30 [ 5044.118207] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5044.118519] obd_mod_free+0x46/0x140 [ptlrpc] [ 5044.118710] mdc_commit_open+0x91/0xb0 [mdc] [ 5044.118745] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5044.118934] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5044.125098] ? do_raw_spin_unlock+0x75/0x190 [ 5044.125120] ? _raw_spin_unlock+0x12/0x30 [ 5044.125190] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5044.125220] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5044.125249] ll_atomic_open+0x36a/0xd70 [lustre] [ 5044.125900] lookup_open+0x356/0xb40 [ 5044.125910] do_last+0x313/0xfc0 [ 5044.125915] ? nd_jump_root+0xe5/0x160 [ 5044.125919] ? path_init+0x437/0x520 [ 5044.125924] path_openat+0xf7/0x500 [ 5044.125929] ? do_raw_spin_unlock+0x75/0x190 [ 5044.125934] do_filp_open+0x99/0x140 [ 5044.125940] ? getname_flags+0x6e/0x330 [ 5044.125945] ? __check_object_size+0xff/0x256 [ 5044.125953] ? do_raw_spin_unlock+0x75/0x190 [ 5044.125957] ? _raw_spin_unlock+0x12/0x30 [ 5044.125961] do_sys_openat2+0x2b4/0x410 [ 5044.125966] do_sys_open+0x73/0xa0 [ 5044.125971] __x64_sys_openat+0x24/0x30 [ 5044.125975] do_syscall_64+0xc1/0x440 [ 5044.125982] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5044.125988] RIP: 0033:0x7f1a63983332 [ 5044.126010] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5044.126015] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5044.126021] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5044.126023] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5044.126026] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5044.126028] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5044.126030] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5044.497350] after_reply+0xe2c/0x1780 [ptlrpc] [ 5044.500770] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5044.508450] ? __next_timer_interrupt+0x160/0x160 [ 5044.513771] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5044.519603] ? do_wait_intr+0xf0/0xf0 [ 5044.522758] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5044.529673] kthread+0x1d1/0x200 [ 5044.532773] ? set_kthread_struct+0x70/0x70 [ 5044.537836] ret_from_fork+0x1f/0x30 [ 5046.240062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3023s! [ 5046.247178] Showing busy workqueues and worker pools: [ 5046.250816] workqueue events: flags=0x0 [ 5046.252829] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5046.256127] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5046.261031] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5046.264444] pending: sec_gc_main [ptlrpc] [ 5046.268591] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5046.273941] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5046.278652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5046.284524] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5046.288512] workqueue events_long: flags=0x0 [ 5046.292632] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5046.297453] pending: lu_objects_flush [obdclass] [ 5046.302672] workqueue events_unbound: flags=0x2 [ 5046.307622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5046.310498] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5046.315189] workqueue events_power_efficient: flags=0x80 [ 5046.317704] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5046.323194] pending: neigh_periodic_work, do_cache_clean [ 5046.326679] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5046.333189] pending: neigh_periodic_work, check_lifetime [ 5046.341027] workqueue mm_percpu_wq: flags=0x8 [ 5046.343091] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5046.346510] pending: vmstat_update [ 5046.348436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5046.350779] pending: vmstat_update [ 5046.356777] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5046.361418] pending: vmstat_update [ 5046.365125] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5046.367957] pending: vmstat_update [ 5046.370055] workqueue writeback: flags=0x4a [ 5046.372837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5046.377311] pending: wb_workfn, wb_workfn [ 5046.380029] workqueue xprtiod: flags=0xa [ 5046.382718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5046.392864] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5046.401204] workqueue ipv6_addrconf: flags=0x40008 [ 5046.406332] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5046.412254] pending: addrconf_verify_work [ 5046.417147] workqueue knbd0-recv: flags=0x1a [ 5046.418940] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5046.424119] in-flight: 647:recv_work [ 5046.425936] workqueue ptlrpc_pinger: flags=0x2 [ 5046.428789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5046.434797] pending: ptlrpc_pinger_main [ptlrpc] [ 5046.439536] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5036s workers=2 idle: 652 [ 5048.062222] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5048.081605] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5048.146395] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5048.154997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5048.162920] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5048.170779] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5048.194185] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5048.203367] RAX: 0000000000001985 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5048.210467] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5048.222030] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5048.228867] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5048.235250] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5048.240730] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5048.245155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5048.247948] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5048.251194] Call Trace: [ 5048.252172] [ 5048.254152] ? show_regs.cold.9+0x22/0x2f [ 5048.259110] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5048.264745] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5048.268558] ? watchdog+0x40/0x40 [ 5048.271548] ? hrtimer_interrupt+0x128/0x2c0 [ 5048.276370] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5048.280237] ? apic_timer_interrupt+0xf/0x20 [ 5048.282667] [ 5048.286712] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5048.292547] do_raw_spin_lock+0x177/0x190 [ 5048.294909] _raw_spin_lock+0x21/0x30 [ 5048.296961] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5048.305213] ? do_raw_spin_unlock+0x75/0x190 [ 5048.312051] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5048.317687] ? do_wait_intr+0xf0/0xf0 [ 5048.320713] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5048.324607] mdc_close+0x53b/0xd70 [mdc] [ 5048.327725] lmv_close+0x18d/0x490 [lmv] [ 5048.332204] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5048.335401] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5048.337550] ll_md_close+0x274/0x800 [lustre] [ 5048.340272] ll_file_release+0x128/0xc50 [lustre] [ 5048.342067] __fput+0x10d/0x440 [ 5048.344983] ____fput+0x12/0x20 [ 5048.348803] task_work_run+0xc6/0x110 [ 5048.353419] exit_to_usermode_loop+0x1dd/0x1f0 [ 5048.355432] do_syscall_64+0x3d6/0x440 [ 5048.357093] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5048.359663] RIP: 0033:0x7f1a63983cd8 [ 5048.362389] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5048.383738] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5048.396735] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5048.410276] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5048.423685] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5048.427327] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5048.430423] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5068.057031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5068.063117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5068.095219] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5068.101496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5068.107136] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5068.109804] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5068.122430] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5068.126559] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5068.131578] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5068.136256] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5068.140361] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5068.144690] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5068.148758] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5068.154948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5068.164267] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5068.175790] Call Trace: [ 5068.176801] [ 5068.177596] ? show_regs.cold.9+0x22/0x2f [ 5068.187258] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5068.193271] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5068.195023] ? watchdog+0x40/0x40 [ 5068.198724] ? hrtimer_interrupt+0x128/0x2c0 [ 5068.202749] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5068.206413] ? apic_timer_interrupt+0xf/0x20 [ 5068.211356] [ 5068.212810] ? native_safe_halt+0x2e/0x50 [ 5068.215638] kvm_wait+0xcd/0xe0 [ 5068.217754] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5068.224306] do_raw_spin_lock+0x177/0x190 [ 5068.234397] _raw_spin_lock+0x21/0x30 [ 5068.249182] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5068.257205] ? do_raw_spin_unlock+0x75/0x190 [ 5068.272322] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5068.281678] ? do_wait_intr+0xf0/0xf0 [ 5068.295300] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5068.313607] mdc_close+0x53b/0xd70 [mdc] [ 5068.318390] lmv_close+0x18d/0x490 [lmv] [ 5068.321448] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5068.325202] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5068.328342] ll_md_close+0x274/0x800 [lustre] [ 5068.332748] ll_file_release+0x128/0xc50 [lustre] [ 5068.336677] __fput+0x10d/0x440 [ 5068.341188] ____fput+0x12/0x20 [ 5068.352448] task_work_run+0xc6/0x110 [ 5068.356616] exit_to_usermode_loop+0x1dd/0x1f0 [ 5068.363959] do_syscall_64+0x3d6/0x440 [ 5068.368949] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5068.375271] RIP: 0033:0x7f1a63983cd8 [ 5068.382765] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5068.412448] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5068.421663] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5068.430573] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5068.438664] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5068.446736] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5068.452754] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5072.055382] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5072.059508] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5072.066020] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5072.066097] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5072.066104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5072.066107] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5072.066122] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5072.066126] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5072.066130] RAX: 0000000000003d09 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5072.066132] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5072.066135] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5072.066137] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5072.066139] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5072.066142] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5072.066145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5072.066147] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5072.066156] Call Trace: [ 5072.066161] [ 5072.066166] ? show_regs.cold.9+0x22/0x2f [ 5072.066178] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5072.066184] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5072.066202] ? watchdog+0x40/0x40 [ 5072.066207] ? hrtimer_interrupt+0x128/0x2c0 [ 5072.066211] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5072.066221] ? apic_timer_interrupt+0xf/0x20 [ 5072.066226] [ 5072.066229] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5072.066235] do_raw_spin_lock+0x177/0x190 [ 5072.066240] _raw_spin_lock+0x21/0x30 [ 5072.066245] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5072.067058] ? do_raw_spin_unlock+0x75/0x190 [ 5072.067065] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5072.067573] ? do_wait_intr+0xf0/0xf0 [ 5072.067582] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5072.068661] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5072.080275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5072.080362] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5072.080370] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5072.080373] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5072.080387] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5072.080391] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5072.080397] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5072.080410] RDX: 00000000000027e4 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5072.080412] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5072.080415] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5072.080418] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5072.080421] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5072.080423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5072.080426] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5072.080435] Call Trace: [ 5072.080440] [ 5072.080446] ? show_regs.cold.9+0x22/0x2f [ 5072.080456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5072.080463] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5072.080467] ? watchdog+0x40/0x40 [ 5072.080471] ? hrtimer_interrupt+0x128/0x2c0 [ 5072.080475] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5072.080485] ? apic_timer_interrupt+0xf/0x20 [ 5072.080490] [ 5072.080493] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5072.080498] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5072.080503] do_raw_spin_lock+0x177/0x190 [ 5072.080508] _raw_spin_lock+0x21/0x30 [ 5072.080513] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5072.081056] obd_mod_free+0x46/0x140 [ptlrpc] [ 5072.081253] mdc_commit_open+0x91/0xb0 [mdc] [ 5072.081289] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5072.082337] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5072.082581] after_reply+0xe2c/0x1780 [ptlrpc] [ 5072.083115] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5072.083307] ? __next_timer_interrupt+0x160/0x160 [ 5072.083318] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5072.147266] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5072.147818] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5072.148105] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5072.148146] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5072.148177] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5072.148424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5072.148518] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5072.149397] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5072.149428] lmv_intent_open+0x338/0x1150 [lmv] [ 5072.149459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5072.149553] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5072.149850] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5072.168501] ? do_wait_intr+0xf0/0xf0 [ 5072.168526] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5072.169139] kthread+0x1d1/0x200 [ 5072.169146] ? set_kthread_struct+0x70/0x70 [ 5072.169151] ret_from_fork+0x1f/0x30 [ 5072.796335] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5072.797638] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5072.801238] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5072.804237] ? do_raw_spin_unlock+0x75/0x190 [ 5072.806662] ? _raw_spin_unlock+0x12/0x30 [ 5072.808887] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5072.810869] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5072.816157] ll_atomic_open+0x36a/0xd70 [lustre] [ 5072.817316] lookup_open+0x356/0xb40 [ 5072.822607] do_last+0x313/0xfc0 [ 5072.823413] ? nd_jump_root+0xe5/0x160 [ 5072.825265] ? path_init+0x437/0x520 [ 5072.829257] path_openat+0xf7/0x500 [ 5072.830238] ? do_raw_spin_unlock+0x75/0x190 [ 5072.836903] do_filp_open+0x99/0x140 [ 5072.837817] ? getname_flags+0x6e/0x330 [ 5072.838659] ? __check_object_size+0xff/0x256 [ 5072.839625] ? do_raw_spin_unlock+0x75/0x190 [ 5072.849287] ? _raw_spin_unlock+0x12/0x30 [ 5072.852261] do_sys_openat2+0x2b4/0x410 [ 5072.856748] do_sys_open+0x73/0xa0 [ 5072.860203] __x64_sys_openat+0x24/0x30 [ 5072.865367] do_syscall_64+0xc1/0x440 [ 5072.870257] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5072.876949] RIP: 0033:0x7f1a63983332 [ 5072.878318] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5072.889908] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5072.898221] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5072.904910] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5072.908130] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5072.912269] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5072.914177] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5076.057382] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5076.061014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5076.104951] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5076.113829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5076.127939] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5076.131025] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5076.141555] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5076.145498] RAX: 000000000000783b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5076.149555] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5076.152587] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5076.156856] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5076.160345] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5076.164212] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5076.168064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5076.171360] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5076.174500] Call Trace: [ 5076.175671] [ 5076.176640] ? show_regs.cold.9+0x22/0x2f [ 5076.178680] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5076.180685] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5076.182752] ? watchdog+0x40/0x40 [ 5076.184243] ? hrtimer_interrupt+0x128/0x2c0 [ 5076.186676] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5076.190938] ? apic_timer_interrupt+0xf/0x20 [ 5076.193349] [ 5076.194280] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5076.196978] do_raw_spin_lock+0x177/0x190 [ 5076.198565] _raw_spin_lock+0x21/0x30 [ 5076.201252] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5076.203915] ? do_raw_spin_unlock+0x75/0x190 [ 5076.205825] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5076.208505] ? do_wait_intr+0xf0/0xf0 [ 5076.210369] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5076.213767] mdc_close+0x53b/0xd70 [mdc] [ 5076.217935] lmv_close+0x18d/0x490 [lmv] [ 5076.222632] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5076.228489] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5076.230918] ll_md_close+0x274/0x800 [lustre] [ 5076.232924] ll_file_release+0x128/0xc50 [lustre] [ 5076.235064] __fput+0x10d/0x440 [ 5076.236810] ____fput+0x12/0x20 [ 5076.242971] task_work_run+0xc6/0x110 [ 5076.247849] exit_to_usermode_loop+0x1dd/0x1f0 [ 5076.252606] do_syscall_64+0x3d6/0x440 [ 5076.256381] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5076.260251] RIP: 0033:0x7f1a63983cd8 [ 5076.265260] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5076.279508] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5076.287185] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5076.293604] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5076.300876] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5076.304981] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5076.310108] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5076.448059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3053s! [ 5076.452643] Showing busy workqueues and worker pools: [ 5076.455635] workqueue events: flags=0x0 [ 5076.457783] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5076.461096] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5076.465944] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5076.469339] pending: sec_gc_main [ptlrpc] [ 5076.471411] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5076.474931] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5076.479569] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5076.482767] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5076.486823] workqueue events_long: flags=0x0 [ 5076.488244] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5076.490678] pending: lu_objects_flush [obdclass] [ 5076.492573] workqueue events_unbound: flags=0x2 [ 5076.494140] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5076.496227] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5076.500156] workqueue events_power_efficient: flags=0x80 [ 5076.502356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5076.505007] pending: neigh_periodic_work, do_cache_clean [ 5076.507041] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5076.509122] pending: neigh_periodic_work, check_lifetime [ 5076.511289] workqueue mm_percpu_wq: flags=0x8 [ 5076.512978] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5076.515227] pending: vmstat_update [ 5076.516599] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5076.519292] pending: vmstat_update [ 5076.520548] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5076.529050] pending: vmstat_update [ 5076.531522] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5076.534178] pending: vmstat_update [ 5076.536166] workqueue writeback: flags=0x4a [ 5076.537429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5076.539938] pending: wb_workfn, wb_workfn [ 5076.541763] workqueue xprtiod: flags=0xa [ 5076.543086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5076.545592] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5076.548636] workqueue ipv6_addrconf: flags=0x40008 [ 5076.550270] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5076.552922] pending: addrconf_verify_work [ 5076.554658] workqueue knbd0-recv: flags=0x1a [ 5076.556503] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5076.558817] in-flight: 647:recv_work [ 5076.560502] workqueue ptlrpc_pinger: flags=0x2 [ 5076.562344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5076.564155] pending: ptlrpc_pinger_main [ptlrpc] [ 5076.566260] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5066s workers=2 idle: 652 [ 5096.057467] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 5096.068069] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5096.118906] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5096.133075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5096.137031] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5096.139841] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5096.153248] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5096.158827] RAX: 000000000000550e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5096.163202] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5096.167425] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5096.171390] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5096.178968] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5096.187949] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5096.193168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5096.196775] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5096.200567] Call Trace: [ 5096.201553] [ 5096.202339] ? show_regs.cold.9+0x22/0x2f [ 5096.206942] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5096.213345] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5096.215719] ? watchdog+0x40/0x40 [ 5096.217231] ? hrtimer_interrupt+0x128/0x2c0 [ 5096.219274] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5096.222281] ? apic_timer_interrupt+0xf/0x20 [ 5096.225880] [ 5096.227896] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5096.231876] do_raw_spin_lock+0x177/0x190 [ 5096.234227] _raw_spin_lock+0x21/0x30 [ 5096.235920] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5096.239912] ? do_raw_spin_unlock+0x75/0x190 [ 5096.242179] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5096.244836] ? do_wait_intr+0xf0/0xf0 [ 5096.246625] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5096.249307] mdc_close+0x53b/0xd70 [mdc] [ 5096.251254] lmv_close+0x18d/0x490 [lmv] [ 5096.253273] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5096.257848] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5096.263903] ll_md_close+0x274/0x800 [lustre] [ 5096.266785] ll_file_release+0x128/0xc50 [lustre] [ 5096.269641] __fput+0x10d/0x440 [ 5096.270821] ____fput+0x12/0x20 [ 5096.277037] task_work_run+0xc6/0x110 [ 5096.279118] exit_to_usermode_loop+0x1dd/0x1f0 [ 5096.281350] do_syscall_64+0x3d6/0x440 [ 5096.282750] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5096.285746] RIP: 0033:0x7f1a63983cd8 [ 5096.291699] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5096.314881] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5096.320997] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5096.327770] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5096.331795] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5096.335889] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5096.340065] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5100.058732] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5100.067997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5100.101695] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5100.116888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5100.123765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5100.129197] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5100.150695] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5100.157158] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5100.162303] RDX: 0000000000002954 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5100.174551] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5100.184774] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5100.192211] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5100.198445] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5100.204538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5100.207140] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5100.210645] Call Trace: [ 5100.212432] [ 5100.217625] ? show_regs.cold.9+0x22/0x2f [ 5100.220271] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5100.230779] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5100.234344] ? watchdog+0x40/0x40 [ 5100.238381] ? hrtimer_interrupt+0x128/0x2c0 [ 5100.240632] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5100.248378] ? apic_timer_interrupt+0xf/0x20 [ 5100.252820] [ 5100.255130] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5100.260232] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5100.265117] do_raw_spin_lock+0x177/0x190 [ 5100.266942] _raw_spin_lock+0x21/0x30 [ 5100.270912] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5100.279773] obd_mod_free+0x46/0x140 [ptlrpc] [ 5100.282498] mdc_commit_open+0x91/0xb0 [mdc] [ 5100.283923] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5100.287533] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5100.291150] after_reply+0xe2c/0x1780 [ptlrpc] [ 5100.293361] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5100.296139] ? __next_timer_interrupt+0x160/0x160 [ 5100.297717] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5100.300146] ? do_wait_intr+0xf0/0xf0 [ 5100.301802] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5100.305316] kthread+0x1d1/0x200 [ 5100.307106] ? set_kthread_struct+0x70/0x70 [ 5100.310030] ret_from_fork+0x1f/0x30 [ 5104.055189] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5104.056093] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5104.067232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [ 5104.071639] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5104.071827] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5104.071834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5104.071837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5104.071851] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5104.071856] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5104.071860] RAX: 0000000000002e96 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5104.071864] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5104.074988] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel [ 5104.101930] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5104.101941] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5104.101947] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5104.101950] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5104.101954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5104.101956] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5104.101964] Call Trace: [ 5104.101971] [ 5104.101978] ? show_regs.cold.9+0x22/0x2f [ 5104.101992] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5104.102003] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5104.102007] ? watchdog+0x40/0x40 [ 5104.102012] ? hrtimer_interrupt+0x128/0x2c0 [ 5104.102016] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5104.102024] ? apic_timer_interrupt+0xf/0x20 [ 5104.102028] [ 5104.102031] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5104.102040] do_raw_spin_lock+0x177/0x190 [ 5104.102045] _raw_spin_lock+0x21/0x30 [ 5104.102050] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5104.102599] ? do_raw_spin_unlock+0x75/0x190 [ 5104.102607] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5104.102826] ? do_wait_intr+0xf0/0xf0 [ 5104.102834] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5104.103036] mdc_close+0x53b/0xd70 [mdc] [ 5104.103079] lmv_close+0x18d/0x490 [lmv] [ 5104.103114] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5104.103246] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5104.109040] ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5104.109068] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5104.109075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5104.109078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5104.109092] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5104.109097] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5104.109101] RAX: 0000000000006984 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5104.109103] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5104.109106] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5104.109108] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5104.109110] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5104.109114] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5104.109116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5104.109118] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5104.109127] Call Trace: [ 5104.109132] [ 5104.109137] ? show_regs.cold.9+0x22/0x2f [ 5104.109147] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5104.109154] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5104.109158] ? watchdog+0x40/0x40 [ 5104.109162] ? hrtimer_interrupt+0x128/0x2c0 [ 5104.109166] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5104.109174] ? apic_timer_interrupt+0xf/0x20 [ 5104.109179] [ 5104.109181] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5104.109186] do_raw_spin_lock+0x177/0x190 [ 5104.109191] _raw_spin_lock+0x21/0x30 [ 5104.109197] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5104.109757] ? do_raw_spin_unlock+0x75/0x190 [ 5104.109763] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5104.109958] ? do_wait_intr+0xf0/0xf0 [ 5104.109966] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5104.110183] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5104.115919] ll_md_close+0x274/0x800 [lustre] [ 5104.116055] ll_file_release+0x128/0xc50 [lustre] [ 5104.116138] __fput+0x10d/0x440 [ 5104.116148] ____fput+0x12/0x20 [ 5104.116152] task_work_run+0xc6/0x110 [ 5104.116162] exit_to_usermode_loop+0x1dd/0x1f0 [ 5104.116171] do_syscall_64+0x3d6/0x440 [ 5104.116176] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5104.116185] RIP: 0033:0x7f1a63983cd8 [ 5104.116194] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5104.116199] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5104.116204] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5104.116206] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5104.116208] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5104.116210] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5104.116212] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5104.463290] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5104.469367] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5104.473037] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5104.478284] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5104.480159] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5104.482332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5104.490753] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5104.497727] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5104.502398] lmv_intent_open+0x338/0x1150 [lmv] [ 5104.504688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5104.508813] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5104.512787] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5104.515198] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5104.519035] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5104.521274] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5104.524689] ? do_raw_spin_unlock+0x75/0x190 [ 5104.530553] ? _raw_spin_unlock+0x12/0x30 [ 5104.535174] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5104.538637] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5104.540194] ll_atomic_open+0x36a/0xd70 [lustre] [ 5104.544088] lookup_open+0x356/0xb40 [ 5104.546509] do_last+0x313/0xfc0 [ 5104.548052] ? nd_jump_root+0xe5/0x160 [ 5104.553933] ? path_init+0x437/0x520 [ 5104.555667] path_openat+0xf7/0x500 [ 5104.558940] ? do_raw_spin_unlock+0x75/0x190 [ 5104.567523] do_filp_open+0x99/0x140 [ 5104.573813] ? getname_flags+0x6e/0x330 [ 5104.576096] ? __check_object_size+0xff/0x256 [ 5104.578201] ? do_raw_spin_unlock+0x75/0x190 [ 5104.579546] ? _raw_spin_unlock+0x12/0x30 [ 5104.582790] do_sys_openat2+0x2b4/0x410 [ 5104.590404] do_sys_open+0x73/0xa0 [ 5104.594468] __x64_sys_openat+0x24/0x30 [ 5104.597682] do_syscall_64+0xc1/0x440 [ 5104.599637] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5104.604206] RIP: 0033:0x7f1a63983332 [ 5104.606053] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5104.615153] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5104.618645] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5104.621601] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5104.623904] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5104.626577] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5104.628946] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5106.662097] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3083s! [ 5106.668123] Showing busy workqueues and worker pools: [ 5106.673380] workqueue events: flags=0x0 [ 5106.675540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5106.683155] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5106.688909] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5106.693148] pending: sec_gc_main [ptlrpc] [ 5106.696085] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5106.702477] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5106.710795] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5106.714959] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5106.725298] workqueue events_long: flags=0x0 [ 5106.727825] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5106.735525] pending: lu_objects_flush [obdclass] [ 5106.739378] workqueue events_unbound: flags=0x2 [ 5106.743794] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5106.751572] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5106.769637] workqueue events_power_efficient: flags=0x80 [ 5106.776990] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5106.786245] pending: neigh_periodic_work, do_cache_clean [ 5106.791086] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5106.796267] pending: neigh_periodic_work, check_lifetime [ 5106.799899] workqueue mm_percpu_wq: flags=0x8 [ 5106.802621] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5106.810336] pending: vmstat_update [ 5106.812390] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5106.819651] pending: vmstat_update [ 5106.825294] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5106.829228] pending: vmstat_update [ 5106.832782] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5106.835002] pending: vmstat_update [ 5106.841370] workqueue writeback: flags=0x4a [ 5106.847083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5106.851238] pending: wb_workfn, wb_workfn [ 5106.856222] workqueue xprtiod: flags=0xa [ 5106.860943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5106.868003] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5106.880368] workqueue ipv6_addrconf: flags=0x40008 [ 5106.886191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5106.898227] pending: addrconf_verify_work [ 5106.902152] workqueue knbd0-recv: flags=0x1a [ 5106.905978] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5106.910799] in-flight: 647:recv_work [ 5106.914148] workqueue ptlrpc_pinger: flags=0x2 [ 5106.920012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5106.925311] pending: ptlrpc_pinger_main [ptlrpc] [ 5106.930117] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5096s workers=2 idle: 652 [ 5124.058438] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 5124.062665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5124.114674] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5124.129946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5124.140875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5124.150302] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5124.167115] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5124.177729] RAX: 0000000000007c0a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5124.189098] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5124.194222] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5124.204699] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5124.209209] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5124.219549] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5124.232123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5124.240806] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5124.247388] Call Trace: [ 5124.249520] [ 5124.250892] ? show_regs.cold.9+0x22/0x2f [ 5124.253629] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5124.260731] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5124.263460] ? watchdog+0x40/0x40 [ 5124.265393] ? hrtimer_interrupt+0x128/0x2c0 [ 5124.275463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5124.278626] ? apic_timer_interrupt+0xf/0x20 [ 5124.281496] [ 5124.283131] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5124.289376] do_raw_spin_lock+0x177/0x190 [ 5124.291632] _raw_spin_lock+0x21/0x30 [ 5124.297369] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5124.301494] ? do_raw_spin_unlock+0x75/0x190 [ 5124.306322] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5124.314195] ? do_wait_intr+0xf0/0xf0 [ 5124.319222] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5124.323902] mdc_close+0x53b/0xd70 [mdc] [ 5124.327923] lmv_close+0x18d/0x490 [lmv] [ 5124.330109] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5124.334176] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5124.342822] ll_md_close+0x274/0x800 [lustre] [ 5124.348792] ll_file_release+0x128/0xc50 [lustre] [ 5124.350802] __fput+0x10d/0x440 [ 5124.352870] ____fput+0x12/0x20 [ 5124.354762] task_work_run+0xc6/0x110 [ 5124.361438] exit_to_usermode_loop+0x1dd/0x1f0 [ 5124.366810] do_syscall_64+0x3d6/0x440 [ 5124.370363] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5124.378475] RIP: 0033:0x7f1a63983cd8 [ 5124.380908] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5124.398729] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5124.409967] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5124.420154] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5124.428046] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5124.436531] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5124.443552] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5128.059373] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5128.069223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5128.122773] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5128.140628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5128.155213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5128.162699] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5128.181338] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5128.188569] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5128.204110] RDX: 0000000000002fe1 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5128.213720] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5128.223452] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5128.226485] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5128.230387] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5128.234672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5128.237757] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5128.244376] Call Trace: [ 5128.251527] [ 5128.257566] ? show_regs.cold.9+0x22/0x2f [ 5128.261033] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5128.263876] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5128.266762] ? watchdog+0x40/0x40 [ 5128.269000] ? hrtimer_interrupt+0x128/0x2c0 [ 5128.272260] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5128.277489] ? apic_timer_interrupt+0xf/0x20 [ 5128.280171] [ 5128.281474] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5128.284410] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5128.287474] do_raw_spin_lock+0x177/0x190 [ 5128.289470] _raw_spin_lock+0x21/0x30 [ 5128.291766] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5128.294229] obd_mod_free+0x46/0x140 [ptlrpc] [ 5128.297721] mdc_commit_open+0x91/0xb0 [mdc] [ 5128.300599] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5128.303655] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5128.306518] after_reply+0xe2c/0x1780 [ptlrpc] [ 5128.309501] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5128.312460] ? __next_timer_interrupt+0x160/0x160 [ 5128.314601] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5128.320454] ? do_wait_intr+0xf0/0xf0 [ 5128.321859] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5128.331652] kthread+0x1d1/0x200 [ 5128.341589] ? set_kthread_struct+0x70/0x70 [ 5128.346102] ret_from_fork+0x1f/0x30 [ 5132.053288] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5132.056127] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5132.056738] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5132.056811] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5132.056818] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5132.056821] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5132.059968] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel [ 5132.096953] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5132.096967] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5132.096974] RAX: 000000000000318b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5132.096976] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5132.096978] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5132.096980] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5132.096982] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5132.096986] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5132.096989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5132.096991] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5132.097001] Call Trace: [ 5132.097006] [ 5132.099078] ? show_regs.cold.9+0x22/0x2f [ 5132.104567] ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5132.104597] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5132.104614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5132.104618] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5132.104634] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5132.104638] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5132.104643] RAX: 0000000000002f4c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5132.104650] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5132.104653] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5132.104655] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5132.104657] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5132.104660] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5132.104662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5132.104664] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5132.104672] Call Trace: [ 5132.104678] [ 5132.104683] ? show_regs.cold.9+0x22/0x2f [ 5132.104693] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5132.104700] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5132.104704] ? watchdog+0x40/0x40 [ 5132.104708] ? hrtimer_interrupt+0x128/0x2c0 [ 5132.104711] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5132.104720] ? apic_timer_interrupt+0xf/0x20 [ 5132.104724] [ 5132.104727] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5132.104733] do_raw_spin_lock+0x177/0x190 [ 5132.104737] _raw_spin_lock+0x21/0x30 [ 5132.104742] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5132.105394] ? do_raw_spin_unlock+0x75/0x190 [ 5132.105401] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5132.105678] ? do_wait_intr+0xf0/0xf0 [ 5132.105687] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5132.106056] mdc_close+0x53b/0xd70 [mdc] [ 5132.106212] lmv_close+0x18d/0x490 [lmv] [ 5132.106245] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5132.106359] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5132.106441] ll_md_close+0x274/0x800 [lustre] [ 5132.112731] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5132.115882] ll_file_release+0x128/0xc50 [lustre] [ 5132.116111] __fput+0x10d/0x440 [ 5132.116121] ____fput+0x12/0x20 [ 5132.116126] task_work_run+0xc6/0x110 [ 5132.116136] exit_to_usermode_loop+0x1dd/0x1f0 [ 5132.116144] do_syscall_64+0x3d6/0x440 [ 5132.116149] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5132.116171] RIP: 0033:0x7f1a63983cd8 [ 5132.116179] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5132.116183] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5132.116189] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5132.116191] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5132.116193] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5132.116196] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5132.116198] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5132.453440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5132.455089] ? watchdog+0x40/0x40 [ 5132.457218] ? hrtimer_interrupt+0x128/0x2c0 [ 5132.459998] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5132.463825] ? apic_timer_interrupt+0xf/0x20 [ 5132.466967] [ 5132.467757] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5132.469596] do_raw_spin_lock+0x177/0x190 [ 5132.472861] _raw_spin_lock+0x21/0x30 [ 5132.474685] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5132.477536] ? do_raw_spin_unlock+0x75/0x190 [ 5132.479786] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5132.486898] ? do_wait_intr+0xf0/0xf0 [ 5132.488565] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5132.491702] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5132.493716] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5132.500760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5132.505300] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5132.508290] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5132.510726] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5132.513022] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5132.518133] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5132.521241] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5132.522956] lmv_intent_open+0x338/0x1150 [lmv] [ 5132.528890] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5132.531990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5132.535119] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5132.537114] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5132.538858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5132.546297] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5132.550765] ? do_raw_spin_unlock+0x75/0x190 [ 5132.555542] ? _raw_spin_unlock+0x12/0x30 [ 5132.559735] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5132.565983] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5132.568452] ll_atomic_open+0x36a/0xd70 [lustre] [ 5132.570680] lookup_open+0x356/0xb40 [ 5132.572379] do_last+0x313/0xfc0 [ 5132.574114] ? nd_jump_root+0xe5/0x160 [ 5132.577243] ? path_init+0x437/0x520 [ 5132.580804] path_openat+0xf7/0x500 [ 5132.582530] ? do_raw_spin_unlock+0x75/0x190 [ 5132.584998] do_filp_open+0x99/0x140 [ 5132.586796] ? getname_flags+0x6e/0x330 [ 5132.588786] ? __check_object_size+0xff/0x256 [ 5132.593342] ? do_raw_spin_unlock+0x75/0x190 [ 5132.595712] ? _raw_spin_unlock+0x12/0x30 [ 5132.597794] do_sys_openat2+0x2b4/0x410 [ 5132.606882] do_sys_open+0x73/0xa0 [ 5132.609710] __x64_sys_openat+0x24/0x30 [ 5132.611029] do_syscall_64+0xc1/0x440 [ 5132.617365] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5132.620239] RIP: 0033:0x7f1a63983332 [ 5132.623263] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5132.641095] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5132.647776] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5132.658397] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5132.661872] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5132.665261] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5132.668626] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5137.387954] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3114s! [ 5137.394462] Showing busy workqueues and worker pools: [ 5137.400632] workqueue events: flags=0x0 [ 5137.404293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5137.411683] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5137.416558] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5137.421235] pending: sec_gc_main [ptlrpc] [ 5137.424771] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5137.430167] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5137.449175] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5137.454028] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5137.466793] workqueue events_long: flags=0x0 [ 5137.469931] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5137.477303] pending: lu_objects_flush [obdclass] [ 5137.484908] workqueue events_unbound: flags=0x2 [ 5137.486439] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5137.495376] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5137.503673] workqueue events_power_efficient: flags=0x80 [ 5137.509315] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5137.517076] pending: neigh_periodic_work, do_cache_clean [ 5137.524746] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5137.531272] pending: neigh_periodic_work, check_lifetime [ 5137.535232] workqueue mm_percpu_wq: flags=0x8 [ 5137.540923] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5137.549439] pending: vmstat_update [ 5137.555143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5137.558555] pending: vmstat_update [ 5137.560887] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5137.572602] pending: vmstat_update [ 5137.575607] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5137.581438] pending: vmstat_update [ 5137.582743] workqueue writeback: flags=0x4a [ 5137.584079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5137.589272] pending: wb_workfn, wb_workfn [ 5137.591044] workqueue xprtiod: flags=0xa [ 5137.593083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5137.596193] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5137.600560] workqueue ipv6_addrconf: flags=0x40008 [ 5137.605783] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5137.611990] pending: addrconf_verify_work [ 5137.614350] workqueue knbd0-recv: flags=0x1a [ 5137.623398] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5137.633375] in-flight: 647:recv_work [ 5137.637898] workqueue ptlrpc_pinger: flags=0x2 [ 5137.647208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5137.657362] pending: ptlrpc_pinger_main [ptlrpc] [ 5137.661685] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5127s workers=2 idle: 652 [ 5142.480030] rcu: INFO: rcu_sched self-detected stall on CPU [ 5142.486207] rcu: 3-...!: (1887939 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 5142.491587] (t=3120084 jiffies g=269633 q=1228) [ 5142.493651] rcu: rcu_sched kthread starved for 3119733 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 5142.497600] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5142.501884] rcu: RCU grace-period kthread stack dump: [ 5142.504027] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5142.507751] Call Trace: [ 5142.508653] __schedule+0x351/0xcb0 [ 5142.510095] schedule+0xc0/0x180 [ 5142.511588] schedule_timeout+0xb4/0x190 [ 5142.513073] ? do_raw_spin_unlock+0x75/0x190 [ 5142.515074] ? __next_timer_interrupt+0x160/0x160 [ 5142.516989] ? prepare_to_swait_event+0xf3/0x140 [ 5142.518998] rcu_gp_kthread+0x8dd/0x15b0 [ 5142.520530] ? force_qs_rnp+0x2c0/0x2c0 [ 5142.522136] kthread+0x1d1/0x200 [ 5142.523858] ? set_kthread_struct+0x70/0x70 [ 5142.525928] ret_from_fork+0x1f/0x30 [ 5142.527992] rcu: Stack dump where RCU GP kthread last ran: [ 5142.531036] NMI backtrace for cpu 3 [ 5142.532742] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5142.540524] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5142.545321] Call Trace: [ 5142.546024] [ 5142.547056] ? dump_stack+0xbb/0x10e [ 5142.548791] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5142.550872] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5142.554198] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5142.556460] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5142.558322] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5142.560817] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 5142.562831] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 5142.565019] ? update_process_times+0x6d/0xc0 [ 5142.566706] ? tick_sched_handle.isra.18+0x27/0x90 [ 5142.568598] ? tick_sched_timer+0x53/0xb0 [ 5142.570271] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5142.571753] ? tick_sched_handle.isra.18+0x90/0x90 [ 5142.573749] ? hrtimer_interrupt+0x128/0x2c0 [ 5142.575879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5142.577683] ? apic_timer_interrupt+0xf/0x20 [ 5142.579425] [ 5142.580197] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5142.582924] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5142.585695] ? do_raw_spin_lock+0x177/0x190 [ 5142.587190] ? _raw_spin_lock+0x21/0x30 [ 5142.589127] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5142.591320] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5142.594008] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5142.596055] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5142.598391] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5142.601140] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5142.603432] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5142.605421] ? __next_timer_interrupt+0x160/0x160 [ 5142.607069] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5142.608777] ? do_wait_intr+0xf0/0xf0 [ 5142.613690] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5142.615885] ? kthread+0x1d1/0x200 [ 5142.617313] ? set_kthread_struct+0x70/0x70 [ 5142.619439] ? ret_from_fork+0x1f/0x30 [ 5142.621135] NMI backtrace for cpu 3 [ 5142.622780] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5142.630324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5142.635531] Call Trace: [ 5142.637082] [ 5142.639244] ? dump_stack+0xbb/0x10e [ 5142.641121] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5142.644241] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5142.646446] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5142.648356] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5142.651147] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5142.653678] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5142.655581] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5142.657798] ? update_process_times+0x6d/0xc0 [ 5142.659760] ? tick_sched_handle.isra.18+0x27/0x90 [ 5142.661877] ? tick_sched_timer+0x53/0xb0 [ 5142.663924] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5142.666883] ? tick_sched_handle.isra.18+0x90/0x90 [ 5142.668732] ? hrtimer_interrupt+0x128/0x2c0 [ 5142.670228] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5142.671987] ? apic_timer_interrupt+0xf/0x20 [ 5142.673511] [ 5142.674306] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5142.676702] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5142.678977] ? do_raw_spin_lock+0x177/0x190 [ 5142.680719] ? _raw_spin_lock+0x21/0x30 [ 5142.682488] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5142.685507] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5142.688101] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5142.690015] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5142.692806] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5142.695236] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5142.697712] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5142.701201] ? __next_timer_interrupt+0x160/0x160 [ 5142.703702] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5142.706005] ? do_wait_intr+0xf0/0xf0 [ 5142.707229] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5142.709025] ? kthread+0x1d1/0x200 [ 5142.710482] ? set_kthread_struct+0x70/0x70 [ 5142.711894] ? ret_from_fork+0x1f/0x30 [ 5152.057010] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 5152.060644] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5152.082570] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5152.088351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5152.093220] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5152.096443] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 5152.107328] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5152.111427] RAX: 00000000000067fe RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5152.114615] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5152.117927] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5152.120355] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5152.123693] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5152.128069] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5152.132269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5152.135482] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5152.137989] Call Trace: [ 5152.138735] [ 5152.139674] ? show_regs.cold.9+0x22/0x2f [ 5152.142544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5152.145770] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5152.148260] ? watchdog+0x40/0x40 [ 5152.152464] ? hrtimer_interrupt+0x128/0x2c0 [ 5152.157588] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5152.166016] ? apic_timer_interrupt+0xf/0x20 [ 5152.167757] [ 5152.168631] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5152.172847] do_raw_spin_lock+0x177/0x190 [ 5152.175085] _raw_spin_lock+0x21/0x30 [ 5152.177111] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5152.179395] ? do_raw_spin_unlock+0x75/0x190 [ 5152.185715] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5152.190640] ? do_wait_intr+0xf0/0xf0 [ 5152.193280] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5152.196180] mdc_close+0x53b/0xd70 [mdc] [ 5152.197530] lmv_close+0x18d/0x490 [lmv] [ 5152.198875] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5152.209117] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5152.212606] ll_md_close+0x274/0x800 [lustre] [ 5152.218495] ll_file_release+0x128/0xc50 [lustre] [ 5152.230092] __fput+0x10d/0x440 [ 5152.231225] ____fput+0x12/0x20 [ 5152.236641] task_work_run+0xc6/0x110 [ 5152.237929] exit_to_usermode_loop+0x1dd/0x1f0 [ 5152.239551] do_syscall_64+0x3d6/0x440 [ 5152.252703] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5152.262269] RIP: 0033:0x7f1a63983cd8 [ 5152.271351] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5152.292016] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5152.298194] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5152.305543] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5152.308690] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5152.316729] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5152.333205] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5160.056019] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5160.062616] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5160.091182] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5160.097680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5160.102572] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5160.105697] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5160.116122] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5160.120679] RAX: 0000000000007985 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5160.124292] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5160.127706] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5160.131350] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5160.136451] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5160.140343] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5160.146623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5160.148654] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5160.151495] Call Trace: [ 5160.152611] [ 5160.154000] ? show_regs.cold.9+0x22/0x2f [ 5160.155504] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5160.160138] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5160.162518] ? watchdog+0x40/0x40 [ 5160.164344] ? hrtimer_interrupt+0x128/0x2c0 [ 5160.166380] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5160.171030] ? apic_timer_interrupt+0xf/0x20 [ 5160.172649] [ 5160.173404] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5160.177349] do_raw_spin_lock+0x177/0x190 [ 5160.179643] _raw_spin_lock+0x21/0x30 [ 5160.181562] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5160.185377] ? do_raw_spin_unlock+0x75/0x190 [ 5160.188105] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5160.190643] ? do_wait_intr+0xf0/0xf0 [ 5160.194986] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5160.200227] mdc_close+0x53b/0xd70 [mdc] [ 5160.202284] lmv_close+0x18d/0x490 [lmv] [ 5160.203773] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5160.207574] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5160.210694] ll_md_close+0x274/0x800 [lustre] [ 5160.213584] ll_file_release+0x128/0xc50 [lustre] [ 5160.218565] __fput+0x10d/0x440 [ 5160.219700] ____fput+0x12/0x20 [ 5160.222601] task_work_run+0xc6/0x110 [ 5160.223946] exit_to_usermode_loop+0x1dd/0x1f0 [ 5160.226419] do_syscall_64+0x3d6/0x440 [ 5160.228514] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5160.235184] RIP: 0033:0x7f1a63983cd8 [ 5160.238991] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5160.252267] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5160.257704] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5160.261469] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5160.265403] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5160.268514] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5160.274304] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5164.053097] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5164.057631] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5164.081697] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5164.086390] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5164.090061] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5164.095027] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5164.105620] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5164.109663] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5164.124085] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5164.140055] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5164.148089] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5164.162137] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5164.172941] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5164.182514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5164.186120] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5164.189696] Call Trace: [ 5164.190554] [ 5164.192627] ? show_regs.cold.9+0x22/0x2f [ 5164.194909] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5164.201737] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5164.207680] ? watchdog+0x40/0x40 [ 5164.209195] ? hrtimer_interrupt+0x128/0x2c0 [ 5164.214585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5164.220834] ? apic_timer_interrupt+0xf/0x20 [ 5164.223206] [ 5164.224031] ? native_safe_halt+0x2e/0x50 [ 5164.229761] kvm_wait+0xcd/0xe0 [ 5164.234220] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5164.239033] do_raw_spin_lock+0x177/0x190 [ 5164.241757] _raw_spin_lock+0x21/0x30 [ 5164.243655] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5164.248054] ? do_raw_spin_unlock+0x75/0x190 [ 5164.252638] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5164.256617] ? do_wait_intr+0xf0/0xf0 [ 5164.261072] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5164.267694] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5164.274059] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5164.279203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5164.281573] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5164.286110] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5164.288476] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5164.299653] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5164.302490] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5164.320529] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5164.323430] lmv_intent_open+0x338/0x1150 [lmv] [ 5164.329673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5164.336786] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5164.344649] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5164.347684] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5164.353301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5164.362817] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5164.367546] ? do_raw_spin_unlock+0x75/0x190 [ 5164.370507] ? _raw_spin_unlock+0x12/0x30 [ 5164.372731] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5164.375650] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5164.377469] ll_atomic_open+0x36a/0xd70 [lustre] [ 5164.381031] lookup_open+0x356/0xb40 [ 5164.382956] do_last+0x313/0xfc0 [ 5164.384680] ? nd_jump_root+0xe5/0x160 [ 5164.386889] ? path_init+0x437/0x520 [ 5164.389085] path_openat+0xf7/0x500 [ 5164.393473] ? do_raw_spin_unlock+0x75/0x190 [ 5164.401833] do_filp_open+0x99/0x140 [ 5164.405207] ? getname_flags+0x6e/0x330 [ 5164.411856] ? __check_object_size+0xff/0x256 [ 5164.416793] ? do_raw_spin_unlock+0x75/0x190 [ 5164.419762] ? _raw_spin_unlock+0x12/0x30 [ 5164.422595] do_sys_openat2+0x2b4/0x410 [ 5164.424766] do_sys_open+0x73/0xa0 [ 5164.426804] __x64_sys_openat+0x24/0x30 [ 5164.429799] do_syscall_64+0xc1/0x440 [ 5164.431963] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5164.438188] RIP: 0033:0x7f1a63983332 [ 5164.440160] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5164.455387] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5164.458798] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5164.464487] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5164.467278] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5164.472741] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5164.483800] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5168.058018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 5168.064968] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5168.096151] BUG: workqueue lockup - pool [ 5168.116531] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5168.116546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5168.116549] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5168.116564] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5168.116569] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 5168.118421] cpus=0-3 flags=0x4 nice=0 stuck for 3145s! [ 5168.118440] Showing busy workqueues and worker pools: [ 5168.118442] workqueue events: flags=0x0 [ 5168.118444] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5168.118454] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5168.118838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5168.118847] pending: sec_gc_main [ptlrpc] [ 5168.119128] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5168.119138] pending: cpu_latency_work [ptlrpc] [ 5168.131343] ORIG_RAX: ffffffffffffff13 [ 5168.131354] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5168.131356] RDX: 0000000000004e74 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5168.131358] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5168.131361] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5168.131363] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5168.131367] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5168.131370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5168.131373] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5168.131383] Call Trace: [ 5168.137873] , nfs4_renew_state [nfsv4], key_garbage_collector [ 5168.138061] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5168.138075] pending: vmstat_shepherd, cpu_latency_work [ptlrpc] [ 5168.140434] [ 5168.140446] ? show_regs.cold.9+0x22/0x2f [ 5168.140461] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5168.140468] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5168.140472] ? watchdog+0x40/0x40 [ 5168.140477] ? hrtimer_interrupt+0x128/0x2c0 [ 5168.140481] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5168.140490] ? apic_timer_interrupt+0xf/0x20 [ 5168.140495] [ 5168.140497] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5168.140505] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5168.140510] do_raw_spin_lock+0x177/0x190 [ 5168.140514] _raw_spin_lock+0x21/0x30 [ 5168.140521] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5168.140795] obd_mod_free+0x46/0x140 [ptlrpc] [ 5168.151829] , osc_grant_work_handler [osc] [ 5168.151934] workqueue events_long: flags=0x0 [ 5168.151941] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5168.151953] pending: lu_objects_flush [obdclass] [ 5168.152156] workqueue events_unbound: flags=0x2 [ 5168.152159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5168.152167] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5168.152191] workqueue events_power_efficient: flags=0x80 [ 5168.152194] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5168.152202] pending: neigh_periodic_work, do_cache_clean [ 5168.152216] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5168.152221] pending: neigh_periodic_work, check_lifetime [ 5168.152302] workqueue mm_percpu_wq: flags=0x8 [ 5168.152305] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5168.152312] pending: vmstat_update [ 5168.152318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5168.152325] pending: vmstat_update [ 5168.152330] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5168.152337] pending: vmstat_update [ 5168.152345] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5168.152352] pending: vmstat_update [ 5168.152358] workqueue writeback: flags=0x4a [ 5168.152360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5168.152366] pending: wb_workfn, wb_workfn [ 5168.152381] workqueue xprtiod: flags=0xa [ 5168.152384] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5168.152390] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5168.152409] workqueue ipv6_addrconf: flags=0x40008 [ 5168.152412] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5168.152418] pending: addrconf_verify_work [ 5168.152430] workqueue knbd0-recv: flags=0x1a [ 5168.152433] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5168.152438] in-flight: 647:recv_work [ 5168.152456] workqueue ptlrpc_pinger: flags=0x2 [ 5168.152458] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5168.152464] pending: ptlrpc_pinger_main [ptlrpc] [ 5168.153081] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5158s workers=2 idle: 652 [ 5168.595278] mdc_commit_open+0x91/0xb0 [mdc] [ 5168.599699] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5168.615592] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5168.632398] after_reply+0xe2c/0x1780 [ptlrpc] [ 5168.647074] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5168.655517] ? __next_timer_interrupt+0x160/0x160 [ 5168.663781] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5168.672583] ? do_wait_intr+0xf0/0xf0 [ 5168.673991] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5168.691445] kthread+0x1d1/0x200 [ 5168.697907] ? set_kthread_struct+0x70/0x70 [ 5168.700477] ret_from_fork+0x1f/0x30 [ 5180.062975] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 5180.068258] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5180.091191] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5180.096637] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5180.101257] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5180.104106] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5180.112385] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5180.115616] RAX: 0000000000001bd1 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5180.118615] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5180.121463] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5180.124776] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5180.127901] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5180.130788] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5180.133843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5180.136339] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5180.139248] Call Trace: [ 5180.140252] [ 5180.140793] ? show_regs.cold.9+0x22/0x2f [ 5180.142393] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5180.144489] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5180.146421] ? watchdog+0x40/0x40 [ 5180.147789] ? hrtimer_interrupt+0x128/0x2c0 [ 5180.149701] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5180.151951] ? apic_timer_interrupt+0xf/0x20 [ 5180.154358] [ 5180.155309] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5180.157623] do_raw_spin_lock+0x177/0x190 [ 5180.159440] _raw_spin_lock+0x21/0x30 [ 5180.161173] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5180.163738] ? do_raw_spin_unlock+0x75/0x190 [ 5180.165612] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5180.167663] ? do_wait_intr+0xf0/0xf0 [ 5180.169373] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5180.171577] mdc_close+0x53b/0xd70 [mdc] [ 5180.173243] lmv_close+0x18d/0x490 [lmv] [ 5180.175071] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5180.177262] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5180.179192] ll_md_close+0x274/0x800 [lustre] [ 5180.181112] ll_file_release+0x128/0xc50 [lustre] [ 5180.183357] __fput+0x10d/0x440 [ 5180.184411] ____fput+0x12/0x20 [ 5180.185804] task_work_run+0xc6/0x110 [ 5180.187291] exit_to_usermode_loop+0x1dd/0x1f0 [ 5180.189189] do_syscall_64+0x3d6/0x440 [ 5180.190748] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5180.192645] RIP: 0033:0x7f1a63983cd8 [ 5180.194167] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5180.200879] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5180.203454] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5180.206071] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5180.208822] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5180.211481] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5180.214245] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5188.056148] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5188.060151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5188.118502] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5188.128920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5188.137816] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5188.145977] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5188.157460] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5188.161560] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5188.170776] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5188.178489] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5188.185124] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5188.189191] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5188.192814] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5188.201261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5188.206508] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5188.212444] Call Trace: [ 5188.214661] [ 5188.216088] ? show_regs.cold.9+0x22/0x2f [ 5188.217856] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5188.220159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5188.222642] ? watchdog+0x40/0x40 [ 5188.224404] ? hrtimer_interrupt+0x128/0x2c0 [ 5188.229304] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5188.232493] ? apic_timer_interrupt+0xf/0x20 [ 5188.234725] [ 5188.235967] ? native_safe_halt+0x2e/0x50 [ 5188.238243] kvm_wait+0xcd/0xe0 [ 5188.239781] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5188.242853] do_raw_spin_lock+0x177/0x190 [ 5188.245258] _raw_spin_lock+0x21/0x30 [ 5188.246917] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5188.250618] ? do_raw_spin_unlock+0x75/0x190 [ 5188.253235] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5188.256920] ? do_wait_intr+0xf0/0xf0 [ 5188.259619] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5188.263340] mdc_close+0x53b/0xd70 [mdc] [ 5188.265304] lmv_close+0x18d/0x490 [lmv] [ 5188.268928] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5188.275783] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5188.281537] ll_md_close+0x274/0x800 [lustre] [ 5188.284673] ll_file_release+0x128/0xc50 [lustre] [ 5188.292707] __fput+0x10d/0x440 [ 5188.296554] ____fput+0x12/0x20 [ 5188.303412] task_work_run+0xc6/0x110 [ 5188.306051] exit_to_usermode_loop+0x1dd/0x1f0 [ 5188.308860] do_syscall_64+0x3d6/0x440 [ 5188.311675] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5188.314668] RIP: 0033:0x7f1a63983cd8 [ 5188.316634] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5188.329848] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5188.335943] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5188.340885] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5188.345253] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5188.348972] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5188.355697] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5196.057629] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 5196.060202] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 5196.063892] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5196.063968] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5196.063975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5196.063978] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5196.063991] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5196.063995] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5196.063999] RAX: 0000000000007e84 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5196.064002] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5196.064004] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5196.064006] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5196.064008] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5196.064012] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5196.064015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5196.064017] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5196.064025] Call Trace: [ 5196.064030] [ 5196.064035] ? show_regs.cold.9+0x22/0x2f [ 5196.064044] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5196.064051] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5196.064055] ? watchdog+0x40/0x40 [ 5196.064059] ? hrtimer_interrupt+0x128/0x2c0 [ 5196.064062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5196.064069] ? apic_timer_interrupt+0xf/0x20 [ 5196.064073] [ 5196.064076] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5196.064082] do_raw_spin_lock+0x177/0x190 [ 5196.064086] _raw_spin_lock+0x21/0x30 [ 5196.064091] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5196.064933] ? do_raw_spin_unlock+0x75/0x190 [ 5196.064939] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5196.072230] Modules linked in: lustre(O) [ 5196.092927] ? do_wait_intr+0xf0/0xf0 [ 5196.092947] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5196.093307] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5196.093788] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5196.094007] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5196.094126] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5196.094162] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5196.094188] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5196.094214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5196.094299] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5196.094790] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5196.094830] lmv_intent_open+0x338/0x1150 [lmv] [ 5196.094862] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5196.095029] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5196.095112] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5196.095135] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5196.095219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5196.095300] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5196.095382] ? do_raw_spin_unlock+0x75/0x190 [ 5196.095390] ? _raw_spin_unlock+0x12/0x30 [ 5196.095403] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5196.095443] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5196.095473] ll_atomic_open+0x36a/0xd70 [lustre] [ 5196.095686] lookup_open+0x356/0xb40 [ 5196.095695] do_last+0x313/0xfc0 [ 5196.095700] ? nd_jump_root+0xe5/0x160 [ 5196.095704] ? path_init+0x437/0x520 [ 5196.095708] path_openat+0xf7/0x500 [ 5196.095713] ? do_raw_spin_unlock+0x75/0x190 [ 5196.095718] do_filp_open+0x99/0x140 [ 5196.103491] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5196.103699] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5196.103706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5196.103709] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5196.103728] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5196.103733] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5196.103738] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5196.103740] RDX: 0000000000004251 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5196.103743] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5196.103745] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5196.103747] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5196.103750] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5196.103752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5196.103755] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5196.103764] Call Trace: [ 5196.103770] [ 5196.103776] ? show_regs.cold.9+0x22/0x2f [ 5196.103785] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5196.103791] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5196.103795] ? watchdog+0x40/0x40 [ 5196.103799] ? hrtimer_interrupt+0x128/0x2c0 [ 5196.103803] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5196.103812] ? apic_timer_interrupt+0xf/0x20 [ 5196.103918] [ 5196.103921] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5196.103926] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5196.103931] do_raw_spin_lock+0x177/0x190 [ 5196.103941] _raw_spin_lock+0x21/0x30 [ 5196.103947] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5196.104295] obd_mod_free+0x46/0x140 [ptlrpc] [ 5196.108573] ? getname_flags+0x6e/0x330 [ 5196.108601] ? __check_object_size+0xff/0x256 [ 5196.108609] ? do_raw_spin_unlock+0x75/0x190 [ 5196.108615] ? _raw_spin_unlock+0x12/0x30 [ 5196.108634] do_sys_openat2+0x2b4/0x410 [ 5196.108641] do_sys_open+0x73/0xa0 [ 5196.108645] __x64_sys_openat+0x24/0x30 [ 5196.108649] do_syscall_64+0xc1/0x440 [ 5196.108661] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5196.108668] RIP: 0033:0x7f1a63983332 [ 5196.108677] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5196.108681] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5196.108686] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5196.108713] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5196.108716] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5196.108718] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5196.108724] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5196.605375] mdc_commit_open+0x91/0xb0 [mdc] [ 5196.614146] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5196.619587] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5196.627000] after_reply+0xe2c/0x1780 [ptlrpc] [ 5196.637783] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5196.646778] ? __next_timer_interrupt+0x160/0x160 [ 5196.660669] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5196.667128] ? do_wait_intr+0xf0/0xf0 [ 5196.672106] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5196.677290] kthread+0x1d1/0x200 [ 5196.688086] ? set_kthread_struct+0x70/0x70 [ 5196.691183] ret_from_fork+0x1f/0x30 [ 5198.305052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3175s! [ 5198.307924] Showing busy workqueues and worker pools: [ 5198.315160] workqueue events: flags=0x0 [ 5198.319615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5198.326749] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5198.330421] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5198.333640] pending: sec_gc_main [ptlrpc] [ 5198.335675] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5198.340920] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5198.350429] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5198.355406] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5198.359857] workqueue events_long: flags=0x0 [ 5198.364127] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5198.368258] pending: lu_objects_flush [obdclass] [ 5198.372896] workqueue events_unbound: flags=0x2 [ 5198.375074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5198.378792] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5198.385131] workqueue events_power_efficient: flags=0x80 [ 5198.398660] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5198.413795] pending: neigh_periodic_work, do_cache_clean [ 5198.421446] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5198.430870] pending: neigh_periodic_work, check_lifetime [ 5198.441443] workqueue mm_percpu_wq: flags=0x8 [ 5198.448871] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5198.460261] pending: vmstat_update [ 5198.462119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5198.467274] pending: vmstat_update [ 5198.473690] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5198.479371] pending: vmstat_update [ 5198.484118] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5198.490113] pending: vmstat_update [ 5198.496202] workqueue writeback: flags=0x4a [ 5198.498850] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5198.502228] pending: wb_workfn, wb_workfn [ 5198.509882] workqueue xprtiod: flags=0xa [ 5198.514197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5198.517055] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5198.522687] workqueue ipv6_addrconf: flags=0x40008 [ 5198.525039] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5198.530224] pending: addrconf_verify_work [ 5198.531425] workqueue knbd0-recv: flags=0x1a [ 5198.536072] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5198.540226] in-flight: 647:recv_work [ 5198.546766] workqueue ptlrpc_pinger: flags=0x2 [ 5198.552011] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5198.559365] pending: ptlrpc_pinger_main [ptlrpc] [ 5198.565414] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5188s workers=2 idle: 652 [ 5208.057020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5208.068617] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5208.106178] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5208.115118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5208.126299] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5208.128529] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5208.163000] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5208.170064] RAX: 0000000000007d8b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5208.174053] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5208.185251] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5208.195712] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5208.208998] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5208.218760] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5208.232297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5208.241142] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5208.244750] Call Trace: [ 5208.245662] [ 5208.246453] ? show_regs.cold.9+0x22/0x2f [ 5208.256151] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5208.260715] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5208.268709] ? watchdog+0x40/0x40 [ 5208.271793] ? hrtimer_interrupt+0x128/0x2c0 [ 5208.276218] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5208.285376] ? apic_timer_interrupt+0xf/0x20 [ 5208.286988] [ 5208.300188] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5208.302092] do_raw_spin_lock+0x177/0x190 [ 5208.315248] _raw_spin_lock+0x21/0x30 [ 5208.323334] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5208.325298] ? do_raw_spin_unlock+0x75/0x190 [ 5208.339500] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5208.358419] ? do_wait_intr+0xf0/0xf0 [ 5208.361667] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5208.377819] mdc_close+0x53b/0xd70 [mdc] [ 5208.386994] lmv_close+0x18d/0x490 [lmv] [ 5208.395203] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5208.407606] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5208.422528] ll_md_close+0x274/0x800 [lustre] [ 5208.428045] ll_file_release+0x128/0xc50 [lustre] [ 5208.429963] __fput+0x10d/0x440 [ 5208.438596] ____fput+0x12/0x20 [ 5208.442019] task_work_run+0xc6/0x110 [ 5208.453283] exit_to_usermode_loop+0x1dd/0x1f0 [ 5208.461072] do_syscall_64+0x3d6/0x440 [ 5208.462389] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5208.471654] RIP: 0033:0x7f1a63983cd8 [ 5208.478227] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5208.512109] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5208.519715] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5208.522944] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5208.527024] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5208.531141] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5208.543903] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5216.059106] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5216.067656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5216.118191] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5216.130514] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5216.141639] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5216.144900] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5216.162190] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5216.170630] RAX: 000000000000260e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5216.173186] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5216.177916] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5216.181966] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5216.188986] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5216.192143] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5216.195273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5216.197397] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5216.199480] Call Trace: [ 5216.200272] [ 5216.201712] ? show_regs.cold.9+0x22/0x2f [ 5216.203587] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5216.205723] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5216.207542] ? watchdog+0x40/0x40 [ 5216.208843] ? hrtimer_interrupt+0x128/0x2c0 [ 5216.210928] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5216.213030] ? apic_timer_interrupt+0xf/0x20 [ 5216.214584] [ 5216.215297] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5216.218691] do_raw_spin_lock+0x177/0x190 [ 5216.220320] _raw_spin_lock+0x21/0x30 [ 5216.222666] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5216.225320] ? do_raw_spin_unlock+0x75/0x190 [ 5216.227795] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5216.230604] ? do_wait_intr+0xf0/0xf0 [ 5216.236156] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5216.239898] mdc_close+0x53b/0xd70 [mdc] [ 5216.246074] lmv_close+0x18d/0x490 [lmv] [ 5216.248685] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5216.254907] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5216.258593] ll_md_close+0x274/0x800 [lustre] [ 5216.260079] ll_file_release+0x128/0xc50 [lustre] [ 5216.266027] __fput+0x10d/0x440 [ 5216.267463] ____fput+0x12/0x20 [ 5216.269283] task_work_run+0xc6/0x110 [ 5216.271865] exit_to_usermode_loop+0x1dd/0x1f0 [ 5216.275070] do_syscall_64+0x3d6/0x440 [ 5216.281896] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5216.288081] RIP: 0033:0x7f1a63983cd8 [ 5216.289317] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5216.303716] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5216.309933] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5216.313060] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5216.315690] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5216.320055] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5216.323884] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5224.057997] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 5224.058008] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 5224.058020] Modules linked in: [ 5224.061490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5224.061568] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5224.061575] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5224.061578] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5224.061594] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5224.061598] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5224.061602] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5224.061604] RDX: 0000000000000a73 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5224.061606] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5224.061608] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5224.061610] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5224.061613] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5224.061616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5224.061618] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5224.061627] Call Trace: [ 5224.061633] [ 5224.061639] ? show_regs.cold.9+0x22/0x2f [ 5224.061648] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5224.061654] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5224.061658] ? watchdog+0x40/0x40 [ 5224.061662] ? hrtimer_interrupt+0x128/0x2c0 [ 5224.061665] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5224.061672] ? apic_timer_interrupt+0xf/0x20 [ 5224.061677] [ 5224.061679] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5224.061684] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5224.061689] do_raw_spin_lock+0x177/0x190 [ 5224.061694] _raw_spin_lock+0x21/0x30 [ 5224.061699] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5224.061958] obd_mod_free+0x46/0x140 [ptlrpc] [ 5224.064439] mdc_commit_open+0x91/0xb0 [mdc] [ 5224.064474] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5224.064721] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5224.064912] after_reply+0xe2c/0x1780 [ptlrpc] [ 5224.065129] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5224.065784] ? __next_timer_interrupt+0x160/0x160 [ 5224.065794] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5224.065994] ? do_wait_intr+0xf0/0xf0 [ 5224.066002] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5224.072123] lustre(O) [ 5224.073675] kthread+0x1d1/0x200 [ 5224.073689] ? set_kthread_struct+0x70/0x70 [ 5224.073695] ret_from_fork+0x1f/0x30 [ 5224.297150] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5224.321031] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5224.325816] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5224.329781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5224.332906] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5224.348112] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5224.352324] RAX: 0000000000007608 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5224.354718] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5224.360648] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5224.363648] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5224.367883] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5224.373643] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5224.379096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5224.381883] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5224.385080] Call Trace: [ 5224.387234] [ 5224.388078] ? show_regs.cold.9+0x22/0x2f [ 5224.392161] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5224.395867] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5224.398118] ? watchdog+0x40/0x40 [ 5224.400715] ? hrtimer_interrupt+0x128/0x2c0 [ 5224.402683] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5224.404849] ? apic_timer_interrupt+0xf/0x20 [ 5224.408164] [ 5224.409732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5224.412414] do_raw_spin_lock+0x177/0x190 [ 5224.414559] _raw_spin_lock+0x21/0x30 [ 5224.416892] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5224.419757] ? do_raw_spin_unlock+0x75/0x190 [ 5224.422181] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5224.428988] ? do_wait_intr+0xf0/0xf0 [ 5224.431395] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5224.436120] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5224.443816] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5224.450102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5224.454988] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5224.457060] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5224.463468] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5224.468967] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5224.477755] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5224.482958] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5224.486975] lmv_intent_open+0x338/0x1150 [lmv] [ 5224.500005] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5224.504160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5224.521774] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5224.525850] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5224.536047] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5224.548579] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5224.564565] ? do_raw_spin_unlock+0x75/0x190 [ 5224.575726] ? _raw_spin_unlock+0x12/0x30 [ 5224.577333] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5224.593524] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5224.605240] ll_atomic_open+0x36a/0xd70 [lustre] [ 5224.617226] lookup_open+0x356/0xb40 [ 5224.623959] do_last+0x313/0xfc0 [ 5224.625231] ? nd_jump_root+0xe5/0x160 [ 5224.626568] ? path_init+0x437/0x520 [ 5224.639551] path_openat+0xf7/0x500 [ 5224.644607] ? do_raw_spin_unlock+0x75/0x190 [ 5224.649537] do_filp_open+0x99/0x140 [ 5224.650521] ? getname_flags+0x6e/0x330 [ 5224.654549] ? __check_object_size+0xff/0x256 [ 5224.663260] ? do_raw_spin_unlock+0x75/0x190 [ 5224.672811] ? _raw_spin_unlock+0x12/0x30 [ 5224.674329] do_sys_openat2+0x2b4/0x410 [ 5224.682339] do_sys_open+0x73/0xa0 [ 5224.691567] __x64_sys_openat+0x24/0x30 [ 5224.694727] do_syscall_64+0xc1/0x440 [ 5224.700501] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5224.705983] RIP: 0033:0x7f1a63983332 [ 5224.707629] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5224.721468] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5224.726336] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5224.729437] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5224.732799] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5224.734806] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5224.737095] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5229.027591] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3206s! [ 5229.037321] Showing busy workqueues and worker pools: [ 5229.041125] workqueue events: flags=0x0 [ 5229.045047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5229.051333] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5229.056920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5229.060107] pending: sec_gc_main [ptlrpc] [ 5229.063179] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5229.074100] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5229.083502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5229.087992] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5229.096681] workqueue events_long: flags=0x0 [ 5229.098411] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5229.106809] pending: lu_objects_flush [obdclass] [ 5229.111917] workqueue events_unbound: flags=0x2 [ 5229.116178] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5229.120987] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5229.130248] workqueue events_power_efficient: flags=0x80 [ 5229.133579] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5229.136062] pending: neigh_periodic_work, do_cache_clean [ 5229.138354] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5229.141156] pending: neigh_periodic_work, check_lifetime [ 5229.144576] workqueue mm_percpu_wq: flags=0x8 [ 5229.147546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5229.151777] pending: vmstat_update [ 5229.153860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5229.156771] pending: vmstat_update [ 5229.158650] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5229.161383] pending: vmstat_update [ 5229.162843] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5229.165959] pending: vmstat_update [ 5229.167700] workqueue writeback: flags=0x4a [ 5229.169653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5229.172491] pending: wb_workfn, wb_workfn [ 5229.174769] workqueue xprtiod: flags=0xa [ 5229.176705] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5229.179742] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5229.186140] workqueue ipv6_addrconf: flags=0x40008 [ 5229.192368] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5229.194867] pending: addrconf_verify_work [ 5229.206119] workqueue knbd0-recv: flags=0x1a [ 5229.209158] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5229.219596] in-flight: 647:recv_work [ 5229.223163] workqueue ptlrpc_pinger: flags=0x2 [ 5229.228707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5229.232054] pending: ptlrpc_pinger_main [ptlrpc] [ 5229.235178] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5219s workers=2 idle: 652 [ 5236.059433] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5236.072620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5236.122816] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5236.137381] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5236.146610] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5236.150861] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5236.178925] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5236.184056] RAX: 0000000000007a8a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5236.190246] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5236.201675] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5236.211664] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5236.230401] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5236.237761] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5236.244890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5236.251427] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5236.261371] Call Trace: [ 5236.264803] [ 5236.267607] ? show_regs.cold.9+0x22/0x2f [ 5236.269958] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5236.276051] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5236.280941] ? watchdog+0x40/0x40 [ 5236.291566] ? hrtimer_interrupt+0x128/0x2c0 [ 5236.294818] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5236.298204] ? apic_timer_interrupt+0xf/0x20 [ 5236.307764] [ 5236.309907] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5236.313466] do_raw_spin_lock+0x177/0x190 [ 5236.319776] _raw_spin_lock+0x21/0x30 [ 5236.326080] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5236.334542] ? do_raw_spin_unlock+0x75/0x190 [ 5236.337907] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5236.343475] ? do_wait_intr+0xf0/0xf0 [ 5236.346910] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5236.355731] mdc_close+0x53b/0xd70 [mdc] [ 5236.362771] lmv_close+0x18d/0x490 [lmv] [ 5236.370403] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5236.374697] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5236.379835] ll_md_close+0x274/0x800 [lustre] [ 5236.385574] ll_file_release+0x128/0xc50 [lustre] [ 5236.396559] __fput+0x10d/0x440 [ 5236.398496] ____fput+0x12/0x20 [ 5236.405307] task_work_run+0xc6/0x110 [ 5236.409370] exit_to_usermode_loop+0x1dd/0x1f0 [ 5236.413977] do_syscall_64+0x3d6/0x440 [ 5236.420193] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5236.427301] RIP: 0033:0x7f1a63983cd8 [ 5236.431510] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5236.448012] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5236.461271] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5236.469859] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5236.474648] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5236.478995] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5236.490505] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5244.057400] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5244.066045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5244.107102] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5244.119412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5244.130542] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5244.136815] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5244.157020] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5244.164492] RAX: 0000000000006b85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5244.170303] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5244.178717] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5244.186250] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5244.192838] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5244.199749] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5244.204333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5244.206964] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5244.210120] Call Trace: [ 5244.211011] [ 5244.212165] ? show_regs.cold.9+0x22/0x2f [ 5244.213996] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5244.218468] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5244.220551] ? watchdog+0x40/0x40 [ 5244.222426] ? hrtimer_interrupt+0x128/0x2c0 [ 5244.224725] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5244.229393] ? apic_timer_interrupt+0xf/0x20 [ 5244.232448] [ 5244.234460] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5244.237365] do_raw_spin_lock+0x177/0x190 [ 5244.240909] _raw_spin_lock+0x21/0x30 [ 5244.244656] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5244.246667] ? do_raw_spin_unlock+0x75/0x190 [ 5244.252695] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5244.259283] ? do_wait_intr+0xf0/0xf0 [ 5244.266547] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5244.269343] mdc_close+0x53b/0xd70 [mdc] [ 5244.271413] lmv_close+0x18d/0x490 [lmv] [ 5244.272986] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5244.279898] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5244.282953] ll_md_close+0x274/0x800 [lustre] [ 5244.285667] ll_file_release+0x128/0xc50 [lustre] [ 5244.287512] __fput+0x10d/0x440 [ 5244.289311] ____fput+0x12/0x20 [ 5244.290859] task_work_run+0xc6/0x110 [ 5244.292489] exit_to_usermode_loop+0x1dd/0x1f0 [ 5244.294626] do_syscall_64+0x3d6/0x440 [ 5244.296556] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5244.298881] RIP: 0033:0x7f1a63983cd8 [ 5244.300667] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5244.313698] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5244.317495] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5244.324400] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5244.327411] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5244.330291] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5244.337754] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5252.060208] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5252.082913] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5252.146744] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5252.173364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5252.180820] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5252.186879] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5252.214027] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5252.217678] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5252.223792] RDX: 0000000000004d3f RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5252.238147] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5252.240713] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5252.257792] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5252.273230] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5252.289818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5252.296043] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5252.301546] Call Trace: [ 5252.306720] [ 5252.309297] ? show_regs.cold.9+0x22/0x2f [ 5252.312056] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5252.317875] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5252.324086] ? watchdog+0x40/0x40 [ 5252.330200] ? hrtimer_interrupt+0x128/0x2c0 [ 5252.332295] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5252.338056] ? apic_timer_interrupt+0xf/0x20 [ 5252.343265] [ 5252.344224] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5252.350078] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5252.355701] do_raw_spin_lock+0x177/0x190 [ 5252.357784] _raw_spin_lock+0x21/0x30 [ 5252.363141] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5252.368558] obd_mod_free+0x46/0x140 [ptlrpc] [ 5252.373525] mdc_commit_open+0x91/0xb0 [mdc] [ 5252.376466] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5252.381444] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5252.386723] after_reply+0xe2c/0x1780 [ptlrpc] [ 5252.389557] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5252.396559] ? __next_timer_interrupt+0x160/0x160 [ 5252.401468] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5252.405237] ? do_wait_intr+0xf0/0xf0 [ 5252.406568] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5252.417551] kthread+0x1d1/0x200 [ 5252.421764] ? set_kthread_struct+0x70/0x70 [ 5252.425156] ret_from_fork+0x1f/0x30 [ 5256.056717] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 5256.065888] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5256.085939] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5256.090651] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5256.094657] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5256.097182] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5256.105312] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5256.108353] RAX: 000000000000553d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5256.111726] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5256.114987] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5256.117683] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5256.120751] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5256.124126] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5256.127634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5256.130169] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5256.133118] Call Trace: [ 5256.134181] [ 5256.134962] ? show_regs.cold.9+0x22/0x2f [ 5256.136662] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5256.138793] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5256.140744] ? watchdog+0x40/0x40 [ 5256.142234] ? hrtimer_interrupt+0x128/0x2c0 [ 5256.144123] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5256.146198] ? apic_timer_interrupt+0xf/0x20 [ 5256.148227] [ 5256.149092] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5256.151431] do_raw_spin_lock+0x177/0x190 [ 5256.153209] _raw_spin_lock+0x21/0x30 [ 5256.154797] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5256.157198] ? do_raw_spin_unlock+0x75/0x190 [ 5256.158844] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5256.161492] ? do_wait_intr+0xf0/0xf0 [ 5256.163224] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5256.165857] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5256.168228] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5256.170676] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5256.173410] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5256.175819] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5256.178054] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5256.180055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5256.182455] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5256.185463] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5256.188098] lmv_intent_open+0x338/0x1150 [lmv] [ 5256.189929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5256.191978] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5256.194306] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5256.196119] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5256.198036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5256.200635] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5256.202432] ? do_raw_spin_unlock+0x75/0x190 [ 5256.204414] ? _raw_spin_unlock+0x12/0x30 [ 5256.206272] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5256.208423] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5256.210884] ll_atomic_open+0x36a/0xd70 [lustre] [ 5256.213316] lookup_open+0x356/0xb40 [ 5256.214808] do_last+0x313/0xfc0 [ 5256.216445] ? nd_jump_root+0xe5/0x160 [ 5256.218109] ? path_init+0x437/0x520 [ 5256.219680] path_openat+0xf7/0x500 [ 5256.221147] ? do_raw_spin_unlock+0x75/0x190 [ 5256.222735] do_filp_open+0x99/0x140 [ 5256.224371] ? getname_flags+0x6e/0x330 [ 5256.226207] ? __check_object_size+0xff/0x256 [ 5256.228198] ? do_raw_spin_unlock+0x75/0x190 [ 5256.230331] ? _raw_spin_unlock+0x12/0x30 [ 5256.232060] do_sys_openat2+0x2b4/0x410 [ 5256.233812] do_sys_open+0x73/0xa0 [ 5256.235421] __x64_sys_openat+0x24/0x30 [ 5256.236984] do_syscall_64+0xc1/0x440 [ 5256.238724] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5256.240752] RIP: 0033:0x7f1a63983332 [ 5256.242523] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5256.250700] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5256.254320] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5256.257502] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5256.260610] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5256.263904] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5256.267231] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5259.748494] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3236s! [ 5259.752145] Showing busy workqueues and worker pools: [ 5259.754851] workqueue events: flags=0x0 [ 5259.756611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5259.761481] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5259.765270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5259.768603] pending: sec_gc_main [ptlrpc] [ 5259.771065] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5259.774456] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5259.779163] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5259.782364] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5259.786846] workqueue events_long: flags=0x0 [ 5259.789268] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5259.792155] pending: lu_objects_flush [obdclass] [ 5259.794667] workqueue events_unbound: flags=0x2 [ 5259.796959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5259.799941] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5259.805515] workqueue events_power_efficient: flags=0x80 [ 5259.807777] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5259.810671] pending: neigh_periodic_work, do_cache_clean [ 5259.813459] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5259.816412] pending: neigh_periodic_work, check_lifetime [ 5259.818894] workqueue mm_percpu_wq: flags=0x8 [ 5259.820780] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5259.823609] pending: vmstat_update [ 5259.825785] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5259.829292] pending: vmstat_update [ 5259.831311] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5259.834677] pending: vmstat_update [ 5259.836600] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5259.840429] pending: vmstat_update [ 5259.842460] workqueue writeback: flags=0x4a [ 5259.844460] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5259.847706] pending: wb_workfn, wb_workfn [ 5259.850176] workqueue xprtiod: flags=0xa [ 5259.852182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5259.855307] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5259.859376] workqueue ipv6_addrconf: flags=0x40008 [ 5259.861782] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5259.864909] pending: addrconf_verify_work [ 5259.867113] workqueue knbd0-recv: flags=0x1a [ 5259.869075] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5259.872305] in-flight: 647:recv_work [ 5259.874166] workqueue ptlrpc_pinger: flags=0x2 [ 5259.875962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5259.878996] pending: ptlrpc_pinger_main [ptlrpc] [ 5259.881504] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5249s workers=2 idle: 652 [ 5264.059101] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5264.064550] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5264.100112] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5264.106114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5264.110094] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5264.115460] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5264.124061] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5264.130090] RAX: 0000000000007f85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5264.133767] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5264.143326] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5264.152108] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5264.156740] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5264.163036] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5264.172751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5264.180522] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5264.187956] Call Trace: [ 5264.189184] [ 5264.190460] ? show_regs.cold.9+0x22/0x2f [ 5264.192770] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5264.195798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5264.197454] ? watchdog+0x40/0x40 [ 5264.199898] ? hrtimer_interrupt+0x128/0x2c0 [ 5264.201152] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5264.203334] ? apic_timer_interrupt+0xf/0x20 [ 5264.207775] [ 5264.208716] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5264.211822] do_raw_spin_lock+0x177/0x190 [ 5264.214748] _raw_spin_lock+0x21/0x30 [ 5264.216708] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5264.219815] ? do_raw_spin_unlock+0x75/0x190 [ 5264.222546] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5264.225541] ? do_wait_intr+0xf0/0xf0 [ 5264.233451] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5264.239167] mdc_close+0x53b/0xd70 [mdc] [ 5264.242441] lmv_close+0x18d/0x490 [lmv] [ 5264.244518] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5264.248510] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5264.255013] ll_md_close+0x274/0x800 [lustre] [ 5264.259198] ll_file_release+0x128/0xc50 [lustre] [ 5264.263791] __fput+0x10d/0x440 [ 5264.265742] ____fput+0x12/0x20 [ 5264.269107] task_work_run+0xc6/0x110 [ 5264.272172] exit_to_usermode_loop+0x1dd/0x1f0 [ 5264.277297] do_syscall_64+0x3d6/0x440 [ 5264.282169] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5264.290792] RIP: 0033:0x7f1a63983cd8 [ 5264.295201] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5264.317672] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5264.331691] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5264.340195] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5264.349814] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5264.355322] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5264.363244] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5272.056811] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5272.068120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5272.127777] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5272.144742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5272.152654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5272.169485] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5272.202463] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5272.219788] RAX: 0000000000005d1b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5272.224200] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5272.235027] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5272.245480] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5272.256555] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5272.270603] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5272.284573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5272.292406] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5272.303986] Call Trace: [ 5272.307597] [ 5272.310735] ? show_regs.cold.9+0x22/0x2f [ 5272.315303] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5272.318040] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5272.322969] ? watchdog+0x40/0x40 [ 5272.326223] ? hrtimer_interrupt+0x128/0x2c0 [ 5272.334484] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5272.343494] ? apic_timer_interrupt+0xf/0x20 [ 5272.347051] [ 5272.350706] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5272.356774] do_raw_spin_lock+0x177/0x190 [ 5272.361268] _raw_spin_lock+0x21/0x30 [ 5272.367060] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5272.374676] ? do_raw_spin_unlock+0x75/0x190 [ 5272.380890] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5272.384678] ? do_wait_intr+0xf0/0xf0 [ 5272.390415] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5272.394673] mdc_close+0x53b/0xd70 [mdc] [ 5272.397820] lmv_close+0x18d/0x490 [lmv] [ 5272.402094] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5272.411119] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5272.415174] ll_md_close+0x274/0x800 [lustre] [ 5272.423692] ll_file_release+0x128/0xc50 [lustre] [ 5272.430527] __fput+0x10d/0x440 [ 5272.434557] ____fput+0x12/0x20 [ 5272.440058] task_work_run+0xc6/0x110 [ 5272.444044] exit_to_usermode_loop+0x1dd/0x1f0 [ 5272.449135] do_syscall_64+0x3d6/0x440 [ 5272.453498] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5272.458909] RIP: 0033:0x7f1a63983cd8 [ 5272.466853] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5272.492585] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5272.499494] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5272.505231] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5272.510355] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5272.519116] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5272.529736] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5280.058046] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5280.070035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5280.114611] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5280.124948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5280.136151] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5280.141395] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5280.158359] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5280.166444] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5280.174554] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5280.181177] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5280.193343] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5280.198391] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5280.205145] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5280.215747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5280.220639] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5280.229460] Call Trace: [ 5280.230959] [ 5280.233569] ? show_regs.cold.9+0x22/0x2f [ 5280.240946] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5280.245664] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5280.249642] ? watchdog+0x40/0x40 [ 5280.253928] ? hrtimer_interrupt+0x128/0x2c0 [ 5280.257885] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5280.264357] ? apic_timer_interrupt+0xf/0x20 [ 5280.267723] [ 5280.269248] ? native_safe_halt+0x2e/0x50 [ 5280.273104] kvm_wait+0xcd/0xe0 [ 5280.275069] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5280.280174] do_raw_spin_lock+0x177/0x190 [ 5280.282746] _raw_spin_lock+0x21/0x30 [ 5280.284182] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5280.287564] obd_mod_free+0x46/0x140 [ptlrpc] [ 5280.291024] mdc_commit_open+0x91/0xb0 [mdc] [ 5280.293220] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5280.301258] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5280.305633] after_reply+0xe2c/0x1780 [ptlrpc] [ 5280.308944] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5280.310467] ? __next_timer_interrupt+0x160/0x160 [ 5280.317692] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5280.323039] ? do_wait_intr+0xf0/0xf0 [ 5280.326367] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5280.330846] kthread+0x1d1/0x200 [ 5280.333212] ? set_kthread_struct+0x70/0x70 [ 5280.335403] ret_from_fork+0x1f/0x30 [ 5284.053046] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 5284.057517] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5284.094859] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5284.099054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5284.102529] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5284.104555] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5284.111763] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5284.114655] RAX: 00000000000055b7 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5284.117330] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5284.119966] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5284.122665] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5284.125342] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5284.128106] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5284.132245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5284.134456] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5284.136838] Call Trace: [ 5284.137754] [ 5284.138561] ? show_regs.cold.9+0x22/0x2f [ 5284.140114] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5284.142014] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5284.143624] ? watchdog+0x40/0x40 [ 5284.144665] ? hrtimer_interrupt+0x128/0x2c0 [ 5284.146550] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5284.148351] ? apic_timer_interrupt+0xf/0x20 [ 5284.150484] [ 5284.151367] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5284.153653] do_raw_spin_lock+0x177/0x190 [ 5284.155360] _raw_spin_lock+0x21/0x30 [ 5284.156614] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5284.158666] ? do_raw_spin_unlock+0x75/0x190 [ 5284.160519] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5284.162625] ? do_wait_intr+0xf0/0xf0 [ 5284.163901] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5284.165861] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5284.167832] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5284.170124] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5284.172001] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5284.173873] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5284.175503] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5284.177355] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5284.179354] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5284.181586] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5284.183419] lmv_intent_open+0x338/0x1150 [lmv] [ 5284.185089] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5284.186882] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5284.188872] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5284.190734] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5284.192287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5284.194227] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5284.195830] ? do_raw_spin_unlock+0x75/0x190 [ 5284.197650] ? _raw_spin_unlock+0x12/0x30 [ 5284.199148] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5284.201245] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5284.202923] ll_atomic_open+0x36a/0xd70 [lustre] [ 5284.204572] lookup_open+0x356/0xb40 [ 5284.206139] do_last+0x313/0xfc0 [ 5284.207338] ? nd_jump_root+0xe5/0x160 [ 5284.208536] ? path_init+0x437/0x520 [ 5284.209786] path_openat+0xf7/0x500 [ 5284.211104] ? do_raw_spin_unlock+0x75/0x190 [ 5284.212783] do_filp_open+0x99/0x140 [ 5284.214132] ? getname_flags+0x6e/0x330 [ 5284.215451] ? __check_object_size+0xff/0x256 [ 5284.216900] ? do_raw_spin_unlock+0x75/0x190 [ 5284.218445] ? _raw_spin_unlock+0x12/0x30 [ 5284.219730] do_sys_openat2+0x2b4/0x410 [ 5284.221159] do_sys_open+0x73/0xa0 [ 5284.222294] __x64_sys_openat+0x24/0x30 [ 5284.223453] do_syscall_64+0xc1/0x440 [ 5284.224579] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5284.226423] RIP: 0033:0x7f1a63983332 [ 5284.227755] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5284.233868] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5284.236351] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5284.238790] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5284.241469] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5284.243863] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5284.246250] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5289.952039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3266s! [ 5289.954865] Showing busy workqueues and worker pools: [ 5289.956494] workqueue events: flags=0x0 [ 5289.957966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5289.960375] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5289.963271] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5289.965380] pending: sec_gc_main [ptlrpc] [ 5289.967161] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5289.969396] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5289.972700] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5289.975183] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5289.978767] workqueue events_long: flags=0x0 [ 5289.980290] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5289.982365] pending: lu_objects_flush [obdclass] [ 5289.984130] workqueue events_unbound: flags=0x2 [ 5289.985595] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5289.987464] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5289.990935] workqueue events_power_efficient: flags=0x80 [ 5289.992577] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5289.994759] pending: neigh_periodic_work, do_cache_clean [ 5289.996246] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5289.998690] pending: neigh_periodic_work, check_lifetime [ 5290.000924] workqueue mm_percpu_wq: flags=0x8 [ 5290.002490] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5290.005114] pending: vmstat_update [ 5290.006537] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5290.009243] pending: vmstat_update [ 5290.011028] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5290.014008] pending: vmstat_update [ 5290.015685] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5290.018465] pending: vmstat_update [ 5290.019919] workqueue writeback: flags=0x4a [ 5290.021425] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5290.023944] pending: wb_workfn, wb_workfn [ 5290.025736] workqueue xprtiod: flags=0xa [ 5290.027315] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5290.029796] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5290.032891] workqueue ipv6_addrconf: flags=0x40008 [ 5290.034900] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5290.037740] pending: addrconf_verify_work [ 5290.039696] workqueue knbd0-recv: flags=0x1a [ 5290.041600] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5290.044489] in-flight: 647:recv_work [ 5290.046219] workqueue ptlrpc_pinger: flags=0x2 [ 5290.048023] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5290.050244] pending: ptlrpc_pinger_main [ptlrpc] [ 5290.052711] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5280s workers=2 idle: 652 [ 5292.057032] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5292.059804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5292.086251] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5292.095616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5292.100233] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5292.103244] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5292.113740] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5292.118679] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5292.122394] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5292.125708] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5292.128579] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5292.132252] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5292.135970] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5292.140227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5292.144084] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5292.147503] Call Trace: [ 5292.148787] [ 5292.149680] ? show_regs.cold.9+0x22/0x2f [ 5292.151814] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5292.154153] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5292.156287] ? watchdog+0x40/0x40 [ 5292.157676] ? hrtimer_interrupt+0x128/0x2c0 [ 5292.159466] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5292.161743] ? apic_timer_interrupt+0xf/0x20 [ 5292.163765] [ 5292.165069] ? native_safe_halt+0x2e/0x50 [ 5292.167574] kvm_wait+0xcd/0xe0 [ 5292.169505] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5292.172066] do_raw_spin_lock+0x177/0x190 [ 5292.173811] _raw_spin_lock+0x21/0x30 [ 5292.176032] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5292.179531] ? do_raw_spin_unlock+0x75/0x190 [ 5292.181496] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5292.184270] ? do_wait_intr+0xf0/0xf0 [ 5292.186135] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5292.188985] mdc_close+0x53b/0xd70 [mdc] [ 5292.191197] lmv_close+0x18d/0x490 [lmv] [ 5292.193198] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5292.196098] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5292.198377] ll_md_close+0x274/0x800 [lustre] [ 5292.200905] ll_file_release+0x128/0xc50 [lustre] [ 5292.203253] __fput+0x10d/0x440 [ 5292.205150] ____fput+0x12/0x20 [ 5292.206660] task_work_run+0xc6/0x110 [ 5292.208387] exit_to_usermode_loop+0x1dd/0x1f0 [ 5292.210398] do_syscall_64+0x3d6/0x440 [ 5292.212846] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5292.215411] RIP: 0033:0x7f1a63983cd8 [ 5292.217446] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5292.226796] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5292.230944] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5292.234886] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5292.239321] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5292.242695] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5292.245950] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5300.057575] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 5300.061324] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5300.099402] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5300.106674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5300.111002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5300.114107] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5300.124437] RSP: 0018:ffffc38542073af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5300.128233] RAX: 0000000000005982 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5300.131260] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5300.134340] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5300.137943] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5300.140891] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5300.144241] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5300.148470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5300.151170] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5300.154919] Call Trace: [ 5300.156174] [ 5300.156965] ? show_regs.cold.9+0x22/0x2f [ 5300.158898] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5300.161297] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5300.163318] ? watchdog+0x40/0x40 [ 5300.164992] ? hrtimer_interrupt+0x128/0x2c0 [ 5300.167239] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5300.169290] ? apic_timer_interrupt+0xf/0x20 [ 5300.171247] [ 5300.172697] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5300.174506] do_raw_spin_lock+0x177/0x190 [ 5300.176587] _raw_spin_lock+0x21/0x30 [ 5300.178309] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5300.180852] ? do_raw_spin_unlock+0x75/0x190 [ 5300.182144] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5300.184284] ? do_wait_intr+0xf0/0xf0 [ 5300.186632] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5300.189152] mdc_close+0x53b/0xd70 [mdc] [ 5300.191004] lmv_close+0x18d/0x490 [lmv] [ 5300.193208] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5300.196799] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5300.199800] ll_md_close+0x274/0x800 [lustre] [ 5300.202790] ll_file_release+0x128/0xc50 [lustre] [ 5300.205769] __fput+0x10d/0x440 [ 5300.207271] ____fput+0x12/0x20 [ 5300.208752] task_work_run+0xc6/0x110 [ 5300.210351] exit_to_usermode_loop+0x1dd/0x1f0 [ 5300.212234] do_syscall_64+0x3d6/0x440 [ 5300.213681] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5300.218206] RIP: 0033:0x7f1a63983cd8 [ 5300.220970] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5300.228966] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5300.233931] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5300.238104] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5300.241972] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5300.246391] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5300.250504] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5308.058729] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5308.063641] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5308.102201] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5308.112371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5308.115898] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5308.120548] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5308.135552] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5308.140412] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5308.144634] RDX: 0000000000007f77 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5308.148971] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5308.152077] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5308.155626] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5308.158879] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5308.162553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5308.164771] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5308.167628] Call Trace: [ 5308.168633] [ 5308.169383] ? show_regs.cold.9+0x22/0x2f [ 5308.170938] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5308.172872] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5308.175630] ? watchdog+0x40/0x40 [ 5308.177271] ? hrtimer_interrupt+0x128/0x2c0 [ 5308.179204] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5308.181412] ? apic_timer_interrupt+0xf/0x20 [ 5308.183267] [ 5308.186204] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5308.189096] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5308.192303] do_raw_spin_lock+0x177/0x190 [ 5308.195113] _raw_spin_lock+0x21/0x30 [ 5308.198608] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5308.202232] obd_mod_free+0x46/0x140 [ptlrpc] [ 5308.205043] mdc_commit_open+0x91/0xb0 [mdc] [ 5308.207817] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5308.210862] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5308.213646] after_reply+0xe2c/0x1780 [ptlrpc] [ 5308.220189] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5308.221885] ? __next_timer_interrupt+0x160/0x160 [ 5308.223102] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5308.226088] ? do_wait_intr+0xf0/0xf0 [ 5308.229006] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5308.234048] kthread+0x1d1/0x200 [ 5308.236819] ? set_kthread_struct+0x70/0x70 [ 5308.238956] ret_from_fork+0x1f/0x30 [ 5316.053320] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5316.061628] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5316.112644] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5316.118252] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5316.121799] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5316.123337] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5316.131625] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5316.134316] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5316.137429] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5316.140606] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5316.144613] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5316.147865] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5316.151958] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5316.156071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5316.159907] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5316.163434] Call Trace: [ 5316.164514] [ 5316.165378] ? show_regs.cold.9+0x22/0x2f [ 5316.167230] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5316.169552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5316.171608] ? watchdog+0x40/0x40 [ 5316.173102] ? hrtimer_interrupt+0x128/0x2c0 [ 5316.174522] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5316.176082] ? apic_timer_interrupt+0xf/0x20 [ 5316.184113] [ 5316.188094] ? native_safe_halt+0x2e/0x50 [ 5316.191541] kvm_wait+0xcd/0xe0 [ 5316.195587] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5316.202067] do_raw_spin_lock+0x177/0x190 [ 5316.205377] _raw_spin_lock+0x21/0x30 [ 5316.207982] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5316.217201] ? do_raw_spin_unlock+0x75/0x190 [ 5316.219253] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5316.224033] ? do_wait_intr+0xf0/0xf0 [ 5316.228116] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5316.233149] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5316.239850] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5316.248172] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5316.252101] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5316.258786] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5316.260778] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5316.264108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5316.268625] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5316.275966] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5316.280500] lmv_intent_open+0x338/0x1150 [lmv] [ 5316.283968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5316.291159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5316.294964] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5316.302636] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5316.309350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5316.311066] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5316.315060] ? do_raw_spin_unlock+0x75/0x190 [ 5316.316986] ? _raw_spin_unlock+0x12/0x30 [ 5316.318176] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5316.326096] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5316.329561] ll_atomic_open+0x36a/0xd70 [lustre] [ 5316.332353] lookup_open+0x356/0xb40 [ 5316.333852] do_last+0x313/0xfc0 [ 5316.336005] ? nd_jump_root+0xe5/0x160 [ 5316.337601] ? path_init+0x437/0x520 [ 5316.339087] path_openat+0xf7/0x500 [ 5316.340693] ? do_raw_spin_unlock+0x75/0x190 [ 5316.346385] do_filp_open+0x99/0x140 [ 5316.350753] ? getname_flags+0x6e/0x330 [ 5316.353177] ? __check_object_size+0xff/0x256 [ 5316.355957] ? do_raw_spin_unlock+0x75/0x190 [ 5316.358040] ? _raw_spin_unlock+0x12/0x30 [ 5316.362978] do_sys_openat2+0x2b4/0x410 [ 5316.367024] do_sys_open+0x73/0xa0 [ 5316.369468] __x64_sys_openat+0x24/0x30 [ 5316.372607] do_syscall_64+0xc1/0x440 [ 5316.373650] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5316.381950] RIP: 0033:0x7f1a63983332 [ 5316.383521] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5316.396454] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5316.400863] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5316.406183] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5316.408335] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5316.412061] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5316.417281] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5320.057011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5320.063755] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5320.093619] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5320.100408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5320.104983] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5320.108667] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5320.120084] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5320.127223] RAX: 0000000000004943 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5320.131737] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5320.136289] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5320.141849] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5320.146405] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5320.150490] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5320.154820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5320.157971] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5320.162466] Call Trace: [ 5320.164209] BUG: workqueue lockup - pool [ 5320.164641] [ 5320.164648] ? show_regs.cold.9+0x22/0x2f [ 5320.164664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5320.164669] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5320.164673] ? watchdog+0x40/0x40 [ 5320.164679] ? hrtimer_interrupt+0x128/0x2c0 [ 5320.164682] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5320.164693] ? apic_timer_interrupt+0xf/0x20 [ 5320.164697] [ 5320.164700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5320.164713] do_raw_spin_lock+0x177/0x190 [ 5320.164718] _raw_spin_lock+0x21/0x30 [ 5320.164723] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5320.165104] ? do_raw_spin_unlock+0x75/0x190 [ 5320.165109] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5320.165395] ? do_wait_intr+0xf0/0xf0 [ 5320.165404] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5320.165672] mdc_close+0x53b/0xd70 [mdc] [ 5320.165727] lmv_close+0x18d/0x490 [lmv] [ 5320.165775] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5320.166017] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5320.166114] ll_md_close+0x274/0x800 [lustre] [ 5320.166197] ll_file_release+0x128/0xc50 [lustre] [ 5320.166281] __fput+0x10d/0x440 [ 5320.166291] ____fput+0x12/0x20 [ 5320.166295] task_work_run+0xc6/0x110 [ 5320.166304] exit_to_usermode_loop+0x1dd/0x1f0 [ 5320.166312] do_syscall_64+0x3d6/0x440 [ 5320.166316] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5320.166389] RIP: 0033:0x7f1a63983cd8 [ 5320.166397] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5320.166401] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5320.166406] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5320.166408] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5320.166411] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5320.166413] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5320.166415] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5320.284168] cpus=0-3 flags=0x4 nice=0 stuck for 3297s! [ 5320.287056] Showing busy workqueues and worker pools: [ 5320.289603] workqueue events: flags=0x0 [ 5320.291832] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5320.296262] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5320.300769] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5320.306389] pending: sec_gc_main [ptlrpc] [ 5320.311128] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5320.316635] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5320.325820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5320.330631] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5320.337113] workqueue events_long: flags=0x0 [ 5320.340080] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5320.344285] pending: lu_objects_flush [obdclass] [ 5320.347601] workqueue events_unbound: flags=0x2 [ 5320.350225] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5320.354189] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5320.360562] workqueue events_power_efficient: flags=0x80 [ 5320.363809] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5320.368526] pending: neigh_periodic_work, do_cache_clean [ 5320.371530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5320.375123] pending: neigh_periodic_work, check_lifetime [ 5320.378613] workqueue mm_percpu_wq: flags=0x8 [ 5320.382035] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5320.385785] pending: vmstat_update [ 5320.389265] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5320.393856] pending: vmstat_update [ 5320.396125] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5320.400028] pending: vmstat_update [ 5320.402192] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5320.406062] pending: vmstat_update [ 5320.408471] workqueue writeback: flags=0x4a [ 5320.411021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5320.414089] pending: wb_workfn, wb_workfn [ 5320.417100] workqueue xprtiod: flags=0xa [ 5320.419465] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5320.422869] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5320.426803] workqueue ipv6_addrconf: flags=0x40008 [ 5320.429269] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5320.432785] pending: addrconf_verify_work [ 5320.435206] workqueue knbd0-recv: flags=0x1a [ 5320.437539] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5320.441226] in-flight: 647:recv_work [ 5320.443926] workqueue ptlrpc_pinger: flags=0x2 [ 5320.447084] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5320.450675] pending: ptlrpc_pinger_main [ptlrpc] [ 5320.455064] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5310s workers=2 idle: 652 [ 5322.483012] rcu: INFO: rcu_sched self-detected stall on CPU [ 5322.490299] rcu: 3-...!: (1995557 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 5322.496090] (t=3300090 jiffies g=269633 q=1279) [ 5322.498448] rcu: rcu_sched kthread starved for 3299737 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 5322.503475] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5322.508543] rcu: RCU grace-period kthread stack dump: [ 5322.511294] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5322.519023] Call Trace: [ 5322.520501] __schedule+0x351/0xcb0 [ 5322.521843] schedule+0xc0/0x180 [ 5322.523434] schedule_timeout+0xb4/0x190 [ 5322.525163] ? do_raw_spin_unlock+0x75/0x190 [ 5322.527419] ? __next_timer_interrupt+0x160/0x160 [ 5322.529667] ? prepare_to_swait_event+0xf3/0x140 [ 5322.531849] rcu_gp_kthread+0x8dd/0x15b0 [ 5322.533701] ? force_qs_rnp+0x2c0/0x2c0 [ 5322.535542] kthread+0x1d1/0x200 [ 5322.536957] ? set_kthread_struct+0x70/0x70 [ 5322.538906] ret_from_fork+0x1f/0x30 [ 5322.540975] rcu: Stack dump where RCU GP kthread last ran: [ 5322.543675] Sending NMI from CPU 3 to CPUs 0: [ 5322.545826] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 5322.546788] NMI backtrace for cpu 3 [ 5322.554942] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5322.564725] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5322.570068] Call Trace: [ 5322.571971] [ 5322.573629] ? dump_stack+0xbb/0x10e [ 5322.576497] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5322.579994] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5322.585117] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5322.587153] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5322.591488] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5322.594575] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5322.596249] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5322.598541] ? update_process_times+0x6d/0xc0 [ 5322.600748] ? tick_sched_handle.isra.18+0x27/0x90 [ 5322.602455] ? tick_sched_timer+0x53/0xb0 [ 5322.604103] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5322.606515] ? tick_sched_handle.isra.18+0x90/0x90 [ 5322.608808] ? hrtimer_interrupt+0x128/0x2c0 [ 5322.610880] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5322.614404] ? apic_timer_interrupt+0xf/0x20 [ 5322.616633] [ 5322.617522] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5322.620299] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5322.622951] ? do_raw_spin_lock+0x177/0x190 [ 5322.624601] ? _raw_spin_lock+0x21/0x30 [ 5322.626426] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5322.629704] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5322.632446] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5322.635010] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5322.638088] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5322.641151] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5322.643801] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5322.647964] ? __next_timer_interrupt+0x160/0x160 [ 5322.650633] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5322.654036] ? do_wait_intr+0xf0/0xf0 [ 5322.656608] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5322.659064] ? kthread+0x1d1/0x200 [ 5322.660860] ? set_kthread_struct+0x70/0x70 [ 5322.662918] ? ret_from_fork+0x1f/0x30 [ 5328.056033] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 5328.061438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5328.102051] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5328.112225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5328.125928] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5328.131587] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5328.149492] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5328.156091] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5328.162921] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5328.172885] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5328.180044] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5328.183940] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5328.187820] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5328.192974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5328.199171] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5328.204600] Call Trace: [ 5328.208713] [ 5328.211711] ? show_regs.cold.9+0x22/0x2f [ 5328.213620] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5328.217496] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5328.221174] ? watchdog+0x40/0x40 [ 5328.225162] ? hrtimer_interrupt+0x128/0x2c0 [ 5328.231874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5328.248074] ? apic_timer_interrupt+0xf/0x20 [ 5328.254725] [ 5328.256766] ? native_safe_halt+0x2e/0x50 [ 5328.262088] kvm_wait+0xcd/0xe0 [ 5328.264578] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5328.273832] do_raw_spin_lock+0x177/0x190 [ 5328.276866] _raw_spin_lock+0x21/0x30 [ 5328.281334] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5328.286512] ? do_raw_spin_unlock+0x75/0x190 [ 5328.293556] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5328.302300] ? do_wait_intr+0xf0/0xf0 [ 5328.305288] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5328.310367] mdc_close+0x53b/0xd70 [mdc] [ 5328.311776] lmv_close+0x18d/0x490 [lmv] [ 5328.314880] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5328.318092] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5328.323268] ll_md_close+0x274/0x800 [lustre] [ 5328.325563] ll_file_release+0x128/0xc50 [lustre] [ 5328.328093] __fput+0x10d/0x440 [ 5328.330099] ____fput+0x12/0x20 [ 5328.336405] task_work_run+0xc6/0x110 [ 5328.340127] exit_to_usermode_loop+0x1dd/0x1f0 [ 5328.344731] do_syscall_64+0x3d6/0x440 [ 5328.347213] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5328.355132] RIP: 0033:0x7f1a63983cd8 [ 5328.359591] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5328.376950] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5328.388404] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5328.392985] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5328.398984] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5328.402986] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5328.406423] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5348.053023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5348.057029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5348.061744] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5348.061824] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5348.061832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5348.061835] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5348.061848] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5348.061853] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5348.061858] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5348.061860] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5348.061862] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5348.061864] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5348.061867] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5348.061870] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5348.061873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5348.061875] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5348.061884] Call Trace: [ 5348.061889] [ 5348.061894] ? show_regs.cold.9+0x22/0x2f [ 5348.061903] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5348.061910] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5348.061914] ? watchdog+0x40/0x40 [ 5348.061918] ? hrtimer_interrupt+0x128/0x2c0 [ 5348.061922] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5348.061928] ? apic_timer_interrupt+0xf/0x20 [ 5348.061933] [ 5348.061936] ? native_safe_halt+0x2e/0x50 [ 5348.061940] kvm_wait+0xcd/0xe0 [ 5348.063286] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5348.063297] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5348.063365] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5348.063371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5348.063374] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5348.063388] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5348.063393] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5348.063397] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5348.063400] RDX: 00000000000072af RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5348.063402] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5348.063404] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5348.063406] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5348.063409] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5348.063412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5348.063414] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5348.063423] Call Trace: [ 5348.063428] [ 5348.063432] ? show_regs.cold.9+0x22/0x2f [ 5348.063441] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5348.063447] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5348.063451] ? watchdog+0x40/0x40 [ 5348.063456] ? hrtimer_interrupt+0x128/0x2c0 [ 5348.063460] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5348.063467] ? apic_timer_interrupt+0xf/0x20 [ 5348.063472] [ 5348.063475] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5348.063479] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5348.063484] do_raw_spin_lock+0x177/0x190 [ 5348.063489] _raw_spin_lock+0x21/0x30 [ 5348.063495] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5348.064800] obd_mod_free+0x46/0x140 [ptlrpc] [ 5348.065008] mdc_commit_open+0x91/0xb0 [mdc] [ 5348.065047] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5348.065267] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5348.065466] after_reply+0xe2c/0x1780 [ptlrpc] [ 5348.066287] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5348.066925] ? __next_timer_interrupt+0x160/0x160 [ 5348.066935] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5348.067161] ? do_wait_intr+0xf0/0xf0 [ 5348.067171] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5348.067370] kthread+0x1d1/0x200 [ 5348.067376] ? set_kthread_struct+0x70/0x70 [ 5348.067379] ret_from_fork+0x1f/0x30 [ 5348.067409] Modules linked in: lustre(O) [ 5348.113054] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5348.118272] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5348.118346] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5348.118354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5348.118358] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5348.118371] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5348.118376] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5348.118381] RAX: 000000000000171f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5348.118384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5348.118386] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5348.118388] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5348.118391] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5348.118395] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5348.118398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5348.118401] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5348.118410] Call Trace: [ 5348.118417] [ 5348.118424] ? show_regs.cold.9+0x22/0x2f [ 5348.118435] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5348.118442] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5348.118447] ? watchdog+0x40/0x40 [ 5348.118451] ? hrtimer_interrupt+0x128/0x2c0 [ 5348.118455] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5348.118462] ? apic_timer_interrupt+0xf/0x20 [ 5348.118467] [ 5348.118470] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5348.118476] do_raw_spin_lock+0x177/0x190 [ 5348.118482] _raw_spin_lock+0x21/0x30 [ 5348.118487] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5348.118753] ? do_raw_spin_unlock+0x75/0x190 [ 5348.118759] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5348.118967] ? do_wait_intr+0xf0/0xf0 [ 5348.118974] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5348.119422] mdc_close+0x53b/0xd70 [mdc] [ 5348.119462] lmv_close+0x18d/0x490 [lmv] [ 5348.119494] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5348.119606] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5348.119690] ll_md_close+0x274/0x800 [lustre] [ 5348.119768] ll_file_release+0x128/0xc50 [lustre] [ 5348.119843] __fput+0x10d/0x440 [ 5348.119851] ____fput+0x12/0x20 [ 5348.119855] task_work_run+0xc6/0x110 [ 5348.119865] exit_to_usermode_loop+0x1dd/0x1f0 [ 5348.119871] do_syscall_64+0x3d6/0x440 [ 5348.119875] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5348.119882] RIP: 0033:0x7f1a63983cd8 [ 5348.119889] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5348.119893] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5348.119898] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5348.119900] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5348.119901] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5348.119903] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5348.119905] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5348.553176] do_raw_spin_lock+0x177/0x190 [ 5348.554628] _raw_spin_lock+0x21/0x30 [ 5348.555620] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5348.557605] ? do_raw_spin_unlock+0x75/0x190 [ 5348.558812] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5348.560600] ? do_wait_intr+0xf0/0xf0 [ 5348.561731] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5348.563248] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5348.564619] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5348.566684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5348.568351] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5348.569713] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5348.571505] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5348.573359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5348.576180] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5348.578945] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5348.584678] lmv_intent_open+0x338/0x1150 [lmv] [ 5348.590603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5348.595387] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5348.600751] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5348.608200] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5348.613833] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5348.616821] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5348.618925] ? do_raw_spin_unlock+0x75/0x190 [ 5348.621859] ? _raw_spin_unlock+0x12/0x30 [ 5348.624774] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5348.628710] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5348.630658] ll_atomic_open+0x36a/0xd70 [lustre] [ 5348.637601] lookup_open+0x356/0xb40 [ 5348.642525] do_last+0x313/0xfc0 [ 5348.646975] ? nd_jump_root+0xe5/0x160 [ 5348.652780] ? path_init+0x437/0x520 [ 5348.655677] path_openat+0xf7/0x500 [ 5348.658447] ? do_raw_spin_unlock+0x75/0x190 [ 5348.662835] do_filp_open+0x99/0x140 [ 5348.665330] ? getname_flags+0x6e/0x330 [ 5348.673550] ? __check_object_size+0xff/0x256 [ 5348.674950] ? do_raw_spin_unlock+0x75/0x190 [ 5348.680995] ? _raw_spin_unlock+0x12/0x30 [ 5348.683928] do_sys_openat2+0x2b4/0x410 [ 5348.686865] do_sys_open+0x73/0xa0 [ 5348.688229] __x64_sys_openat+0x24/0x30 [ 5348.693208] do_syscall_64+0xc1/0x440 [ 5348.694898] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5348.696807] RIP: 0033:0x7f1a63983332 [ 5348.699496] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5348.711675] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5348.716741] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5348.721297] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5348.731240] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5348.736598] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5348.743228] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5350.881070] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3327s! [ 5350.884238] Showing busy workqueues and worker pools: [ 5350.886178] workqueue events: flags=0x0 [ 5350.887742] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5350.891268] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5350.895127] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5350.898619] pending: sec_gc_main [ptlrpc] [ 5350.901215] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5350.904797] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5350.909793] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5350.912662] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5350.916829] workqueue events_long: flags=0x0 [ 5350.918526] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5350.921737] pending: lu_objects_flush [obdclass] [ 5350.923796] workqueue events_unbound: flags=0x2 [ 5350.925499] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5350.927941] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5350.931941] workqueue events_power_efficient: flags=0x80 [ 5350.934721] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5350.937377] pending: neigh_periodic_work, do_cache_clean [ 5350.939729] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5350.943543] pending: neigh_periodic_work, check_lifetime [ 5350.947385] workqueue mm_percpu_wq: flags=0x8 [ 5350.951378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5350.955441] pending: vmstat_update [ 5350.957604] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5350.960559] pending: vmstat_update [ 5350.963350] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5350.967628] pending: vmstat_update [ 5350.968834] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5350.971549] pending: vmstat_update [ 5350.973103] workqueue writeback: flags=0x4a [ 5350.974817] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5350.978269] pending: wb_workfn, wb_workfn [ 5350.980373] workqueue xprtiod: flags=0xa [ 5350.982211] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5350.986014] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5350.992262] workqueue ipv6_addrconf: flags=0x40008 [ 5350.996180] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5351.000127] pending: addrconf_verify_work [ 5351.011346] workqueue knbd0-recv: flags=0x1a [ 5351.012980] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5351.016748] in-flight: 647:recv_work [ 5351.019327] workqueue ptlrpc_pinger: flags=0x2 [ 5351.021668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5351.025019] pending: ptlrpc_pinger_main [ptlrpc] [ 5351.027343] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5341s workers=2 idle: 652 [ 5356.056023] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 5356.062490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5356.098860] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5356.110697] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5356.115472] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5356.117846] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5356.131440] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5356.135743] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5356.140481] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5356.144422] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5356.148649] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5356.154525] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5356.158527] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5356.162565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5356.165750] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5356.169462] Call Trace: [ 5356.170738] [ 5356.172697] ? show_regs.cold.9+0x22/0x2f [ 5356.175833] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5356.180377] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5356.182798] ? watchdog+0x40/0x40 [ 5356.184604] ? hrtimer_interrupt+0x128/0x2c0 [ 5356.186952] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5356.189782] ? apic_timer_interrupt+0xf/0x20 [ 5356.192482] [ 5356.193927] ? native_safe_halt+0x2e/0x50 [ 5356.196452] kvm_wait+0xcd/0xe0 [ 5356.198356] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5356.201581] do_raw_spin_lock+0x177/0x190 [ 5356.203882] _raw_spin_lock+0x21/0x30 [ 5356.205438] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5356.207924] ? do_raw_spin_unlock+0x75/0x190 [ 5356.210646] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5356.213914] ? do_wait_intr+0xf0/0xf0 [ 5356.216346] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5356.219769] mdc_close+0x53b/0xd70 [mdc] [ 5356.223045] lmv_close+0x18d/0x490 [lmv] [ 5356.226899] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5356.233522] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5356.238864] ll_md_close+0x274/0x800 [lustre] [ 5356.241746] ll_file_release+0x128/0xc50 [lustre] [ 5356.245368] __fput+0x10d/0x440 [ 5356.247420] ____fput+0x12/0x20 [ 5356.249428] task_work_run+0xc6/0x110 [ 5356.251535] exit_to_usermode_loop+0x1dd/0x1f0 [ 5356.254535] do_syscall_64+0x3d6/0x440 [ 5356.256566] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5356.259639] RIP: 0033:0x7f1a63983cd8 [ 5356.261754] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5356.271666] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5356.275384] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5356.279681] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5356.289597] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5356.296780] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5356.301521] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5376.054726] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5376.057354] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5376.058370] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5376.058379] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5376.058455] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5376.058460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5376.058463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5376.058477] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5376.058481] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5376.058486] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5376.058488] RDX: 000000000000057b RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5376.058491] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5376.058493] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5376.058496] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5376.058499] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5376.058502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5376.058505] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5376.058512] Call Trace: [ 5376.058518] [ 5376.058523] ? show_regs.cold.9+0x22/0x2f [ 5376.058532] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5376.058539] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5376.058543] ? watchdog+0x40/0x40 [ 5376.058547] ? hrtimer_interrupt+0x128/0x2c0 [ 5376.058551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5376.058559] ? apic_timer_interrupt+0xf/0x20 [ 5376.058564] [ 5376.058566] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5376.058571] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5376.058576] do_raw_spin_lock+0x177/0x190 [ 5376.058580] _raw_spin_lock+0x21/0x30 [ 5376.058637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5376.058698] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5376.058585] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5376.058875] obd_mod_free+0x46/0x140 [ptlrpc] [ 5376.059062] mdc_commit_open+0x91/0xb0 [mdc] [ 5376.059098] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5376.059288] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5376.059502] after_reply+0xe2c/0x1780 [ptlrpc] [ 5376.059718] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5376.059941] ? __next_timer_interrupt+0x160/0x160 [ 5376.059950] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5376.060168] ? do_wait_intr+0xf0/0xf0 [ 5376.060176] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5376.060379] kthread+0x1d1/0x200 [ 5376.060383] ? set_kthread_struct+0x70/0x70 [ 5376.060387] ret_from_fork+0x1f/0x30 [ 5376.064476] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5376.064556] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5376.064563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5376.064566] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5376.064580] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5376.064584] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5376.066245] RAX: 00000000000077a6 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5376.066252] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5376.066254] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5376.066256] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5376.066258] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5376.066262] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5376.066265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5376.066267] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5376.066276] Call Trace: [ 5376.066282] [ 5376.066288] ? show_regs.cold.9+0x22/0x2f [ 5376.066508] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5376.066516] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5376.066520] ? watchdog+0x40/0x40 [ 5376.066525] ? hrtimer_interrupt+0x128/0x2c0 [ 5376.066529] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5376.066538] ? apic_timer_interrupt+0xf/0x20 [ 5376.066543] [ 5376.066546] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5376.066554] do_raw_spin_lock+0x177/0x190 [ 5376.066559] _raw_spin_lock+0x21/0x30 [ 5376.066565] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5376.076908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5376.076919] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5376.076933] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5376.076937] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5376.076943] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5376.076945] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5376.076948] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5376.076951] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5376.076953] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5376.076957] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5376.076960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5376.076962] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5376.076970] Call Trace: [ 5376.076976] [ 5376.076982] ? show_regs.cold.9+0x22/0x2f [ 5376.076994] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5376.077000] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5376.077005] ? watchdog+0x40/0x40 [ 5376.077010] ? hrtimer_interrupt+0x128/0x2c0 [ 5376.077015] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5376.077022] ? apic_timer_interrupt+0xf/0x20 [ 5376.077027] [ 5376.077031] ? native_safe_halt+0x2e/0x50 [ 5376.077034] kvm_wait+0xcd/0xe0 [ 5376.077041] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5376.077049] do_raw_spin_lock+0x177/0x190 [ 5376.077053] _raw_spin_lock+0x21/0x30 [ 5376.077058] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5376.077689] ? do_raw_spin_unlock+0x75/0x190 [ 5376.077696] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5376.077977] ? do_wait_intr+0xf0/0xf0 [ 5376.078115] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5376.078427] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5376.111552] ? do_raw_spin_unlock+0x75/0x190 [ 5376.111574] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5376.112047] ? do_wait_intr+0xf0/0xf0 [ 5376.112055] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5376.112263] mdc_close+0x53b/0xd70 [mdc] [ 5376.112307] lmv_close+0x18d/0x490 [lmv] [ 5376.112339] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5376.112460] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5376.112541] ll_md_close+0x274/0x800 [lustre] [ 5376.112745] ll_file_release+0x128/0xc50 [lustre] [ 5376.112906] __fput+0x10d/0x440 [ 5376.112915] ____fput+0x12/0x20 [ 5376.112919] task_work_run+0xc6/0x110 [ 5376.112926] exit_to_usermode_loop+0x1dd/0x1f0 [ 5376.112933] do_syscall_64+0x3d6/0x440 [ 5376.112937] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5376.112946] RIP: 0033:0x7f1a63983cd8 [ 5376.112953] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5376.112957] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5376.112962] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5376.112964] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5376.112966] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5376.112969] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5376.112971] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5376.484028] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5376.497925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5376.500138] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5376.508609] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5376.511059] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5376.514581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5376.522481] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5376.527520] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5376.534573] lmv_intent_open+0x338/0x1150 [lmv] [ 5376.539514] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5376.541361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5376.547408] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5376.554688] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5376.558161] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5376.560673] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5376.566647] ? do_raw_spin_unlock+0x75/0x190 [ 5376.568536] ? _raw_spin_unlock+0x12/0x30 [ 5376.573344] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5376.580169] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5376.581695] ll_atomic_open+0x36a/0xd70 [lustre] [ 5376.586613] lookup_open+0x356/0xb40 [ 5376.589793] do_last+0x313/0xfc0 [ 5376.590993] ? nd_jump_root+0xe5/0x160 [ 5376.596729] ? path_init+0x437/0x520 [ 5376.598364] path_openat+0xf7/0x500 [ 5376.608151] ? do_raw_spin_unlock+0x75/0x190 [ 5376.614633] do_filp_open+0x99/0x140 [ 5376.620759] ? getname_flags+0x6e/0x330 [ 5376.624141] ? __check_object_size+0xff/0x256 [ 5376.629536] ? do_raw_spin_unlock+0x75/0x190 [ 5376.635063] ? _raw_spin_unlock+0x12/0x30 [ 5376.639344] do_sys_openat2+0x2b4/0x410 [ 5376.644703] do_sys_open+0x73/0xa0 [ 5376.645645] __x64_sys_openat+0x24/0x30 [ 5376.648646] do_syscall_64+0xc1/0x440 [ 5376.654083] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5376.661074] RIP: 0033:0x7f1a63983332 [ 5376.662241] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5376.682157] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5376.690942] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5376.695357] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5376.701645] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5376.711163] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5376.718124] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5381.090820] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3358s! [ 5381.094403] Showing busy workqueues and worker pools: [ 5381.097399] workqueue events: flags=0x0 [ 5381.100356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5381.104407] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5381.109084] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5381.113101] pending: sec_gc_main [ptlrpc] [ 5381.115712] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5381.120566] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5381.131292] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5381.136212] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5381.143869] workqueue events_long: flags=0x0 [ 5381.148563] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5381.152977] pending: lu_objects_flush [obdclass] [ 5381.157079] workqueue events_unbound: flags=0x2 [ 5381.160156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5381.166621] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5381.178880] workqueue events_power_efficient: flags=0x80 [ 5381.180603] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5381.182962] pending: neigh_periodic_work, do_cache_clean [ 5381.185293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5381.188902] pending: neigh_periodic_work, check_lifetime [ 5381.194595] workqueue mm_percpu_wq: flags=0x8 [ 5381.196876] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5381.202145] pending: vmstat_update [ 5381.203852] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5381.209379] pending: vmstat_update [ 5381.210820] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5381.213469] pending: vmstat_update [ 5381.216855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5381.221702] pending: vmstat_update [ 5381.222987] workqueue writeback: flags=0x4a [ 5381.225059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5381.228831] pending: wb_workfn, wb_workfn [ 5381.229960] workqueue xprtiod: flags=0xa [ 5381.233135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5381.237564] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5381.242978] workqueue ipv6_addrconf: flags=0x40008 [ 5381.250541] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5381.255724] pending: addrconf_verify_work [ 5381.258028] workqueue knbd0-recv: flags=0x1a [ 5381.260282] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5381.262670] in-flight: 647:recv_work [ 5381.263718] workqueue ptlrpc_pinger: flags=0x2 [ 5381.266042] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5381.267506] pending: ptlrpc_pinger_main [ptlrpc] [ 5381.269097] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5371s workers=2 idle: 652 [ 5384.056470] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 5384.059123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5384.076956] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5384.082492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5384.086613] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5384.089826] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5384.100423] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5384.104148] RAX: 0000000000001835 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5384.107124] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5384.111341] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5384.116529] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5384.121608] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5384.125169] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5384.129109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5384.131437] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5384.135115] Call Trace: [ 5384.136176] [ 5384.137143] ? show_regs.cold.9+0x22/0x2f [ 5384.139636] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5384.141662] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5384.144109] ? watchdog+0x40/0x40 [ 5384.145854] ? hrtimer_interrupt+0x128/0x2c0 [ 5384.147793] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5384.150435] ? apic_timer_interrupt+0xf/0x20 [ 5384.152449] [ 5384.153418] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5384.156212] do_raw_spin_lock+0x177/0x190 [ 5384.158135] _raw_spin_lock+0x21/0x30 [ 5384.159853] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5384.162277] ? do_raw_spin_unlock+0x75/0x190 [ 5384.164383] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5384.166915] ? do_wait_intr+0xf0/0xf0 [ 5384.168697] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5384.171282] mdc_close+0x53b/0xd70 [mdc] [ 5384.175343] lmv_close+0x18d/0x490 [lmv] [ 5384.177385] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5384.180314] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5384.182886] ll_md_close+0x274/0x800 [lustre] [ 5384.192324] ll_file_release+0x128/0xc50 [lustre] [ 5384.195467] __fput+0x10d/0x440 [ 5384.197531] ____fput+0x12/0x20 [ 5384.201182] task_work_run+0xc6/0x110 [ 5384.203566] exit_to_usermode_loop+0x1dd/0x1f0 [ 5384.205977] do_syscall_64+0x3d6/0x440 [ 5384.207231] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5384.210383] RIP: 0033:0x7f1a63983cd8 [ 5384.213539] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5384.223408] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5384.226887] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5384.230040] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5384.233060] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5384.236487] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5384.238565] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5404.057017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5404.060046] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5404.062089] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5404.077155] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5404.077168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5404.077171] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5404.077239] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5404.077243] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5404.077248] RAX: 0000000000002a2c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5404.077251] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5404.077253] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5404.077254] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5404.077256] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5404.077260] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5404.077262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5404.077264] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5404.077272] Call Trace: [ 5404.077283] [ 5404.077289] ? show_regs.cold.9+0x22/0x2f [ 5404.077297] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5404.077303] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5404.077307] ? watchdog+0x40/0x40 [ 5404.077310] ? hrtimer_interrupt+0x128/0x2c0 [ 5404.077313] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5404.077320] ? apic_timer_interrupt+0xf/0x20 [ 5404.077324] [ 5404.077326] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5404.077332] do_raw_spin_lock+0x177/0x190 [ 5404.077336] _raw_spin_lock+0x21/0x30 [ 5404.077341] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5404.077676] ? do_raw_spin_unlock+0x75/0x190 [ 5404.077738] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5404.078000] ? do_wait_intr+0xf0/0xf0 [ 5404.078009] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5404.078288] mdc_close+0x53b/0xd70 [mdc] [ 5404.078325] lmv_close+0x18d/0x490 [lmv] [ 5404.078346] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5404.078574] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5404.078650] ll_md_close+0x274/0x800 [lustre] [ 5404.078728] ll_file_release+0x128/0xc50 [lustre] [ 5404.078822] __fput+0x10d/0x440 [ 5404.078831] ____fput+0x12/0x20 [ 5404.078836] task_work_run+0xc6/0x110 [ 5404.078845] exit_to_usermode_loop+0x1dd/0x1f0 [ 5404.078851] do_syscall_64+0x3d6/0x440 [ 5404.078855] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5404.078862] RIP: 0033:0x7f1a63983cd8 [ 5404.078878] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5404.078882] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5404.078886] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5404.078888] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5404.078891] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5404.078893] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5404.078899] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5404.445146] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5404.506623] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5404.514064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5404.517933] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5404.521059] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5404.533819] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5404.543601] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5404.550203] RDX: 0000000000004489 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5404.555183] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5404.558232] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5404.563930] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5404.567108] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5404.573667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5404.576398] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5404.580023] Call Trace: [ 5404.581252] [ 5404.582173] ? show_regs.cold.9+0x22/0x2f [ 5404.584058] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5404.586266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5404.588287] ? watchdog+0x40/0x40 [ 5404.589993] ? hrtimer_interrupt+0x128/0x2c0 [ 5404.592149] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5404.594359] ? apic_timer_interrupt+0xf/0x20 [ 5404.596371] [ 5404.597911] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5404.600094] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5404.603240] do_raw_spin_lock+0x177/0x190 [ 5404.605723] _raw_spin_lock+0x21/0x30 [ 5404.607767] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5404.610187] obd_mod_free+0x46/0x140 [ptlrpc] [ 5404.612689] mdc_commit_open+0x91/0xb0 [mdc] [ 5404.614865] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5404.617562] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5404.620140] after_reply+0xe2c/0x1780 [ptlrpc] [ 5404.627124] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5404.631146] ? __next_timer_interrupt+0x160/0x160 [ 5404.634494] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5404.639313] ? do_wait_intr+0xf0/0xf0 [ 5404.642977] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5404.649367] kthread+0x1d1/0x200 [ 5404.651825] ? set_kthread_struct+0x70/0x70 [ 5404.655639] ret_from_fork+0x1f/0x30 [ 5408.053038] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5408.059042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5408.113215] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5408.123389] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5408.133189] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5408.139820] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5408.155971] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5408.161141] RAX: 00000000000039d8 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5408.165067] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5408.170441] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5408.174066] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5408.176992] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5408.181083] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5408.187025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5408.190021] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5408.195085] Call Trace: [ 5408.197735] [ 5408.198534] ? show_regs.cold.9+0x22/0x2f [ 5408.200066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5408.215808] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5408.220998] ? watchdog+0x40/0x40 [ 5408.223272] ? hrtimer_interrupt+0x128/0x2c0 [ 5408.226314] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5408.231101] ? apic_timer_interrupt+0xf/0x20 [ 5408.233622] [ 5408.235917] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5408.238116] do_raw_spin_lock+0x177/0x190 [ 5408.240299] _raw_spin_lock+0x21/0x30 [ 5408.242283] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5408.245430] ? do_raw_spin_unlock+0x75/0x190 [ 5408.247422] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5408.250674] ? do_wait_intr+0xf0/0xf0 [ 5408.253136] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5408.258188] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5408.273406] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5408.281853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5408.289432] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5408.294072] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5408.300624] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5408.305886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5408.316049] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5408.323783] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5408.329361] lmv_intent_open+0x338/0x1150 [lmv] [ 5408.333870] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5408.335720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5408.339052] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5408.346847] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5408.352102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5408.359991] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5408.363460] ? do_raw_spin_unlock+0x75/0x190 [ 5408.365082] ? _raw_spin_unlock+0x12/0x30 [ 5408.368998] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5408.374698] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5408.378953] ll_atomic_open+0x36a/0xd70 [lustre] [ 5408.381506] lookup_open+0x356/0xb40 [ 5408.382896] do_last+0x313/0xfc0 [ 5408.388029] ? nd_jump_root+0xe5/0x160 [ 5408.394075] ? path_init+0x437/0x520 [ 5408.396323] path_openat+0xf7/0x500 [ 5408.400371] ? do_raw_spin_unlock+0x75/0x190 [ 5408.402208] do_filp_open+0x99/0x140 [ 5408.404313] ? getname_flags+0x6e/0x330 [ 5408.408275] ? __check_object_size+0xff/0x256 [ 5408.413011] ? do_raw_spin_unlock+0x75/0x190 [ 5408.417322] ? _raw_spin_unlock+0x12/0x30 [ 5408.419663] do_sys_openat2+0x2b4/0x410 [ 5408.421884] do_sys_open+0x73/0xa0 [ 5408.423184] __x64_sys_openat+0x24/0x30 [ 5408.425745] do_syscall_64+0xc1/0x440 [ 5408.427952] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5408.435210] RIP: 0033:0x7f1a63983332 [ 5408.438643] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5408.454231] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5408.463913] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5408.468223] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5408.476145] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5408.482112] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5408.487464] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5411.302216] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3388s! [ 5411.306691] Showing busy workqueues and worker pools: [ 5411.308589] workqueue events: flags=0x0 [ 5411.315301] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5411.324372] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5411.338105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5411.340432] pending: sec_gc_main [ptlrpc] [ 5411.351542] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5411.365831] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5411.384155] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5411.392267] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5411.401029] workqueue events_long: flags=0x0 [ 5411.404584] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5411.410673] pending: lu_objects_flush [obdclass] [ 5411.414792] workqueue events_unbound: flags=0x2 [ 5411.417287] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5411.422594] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5411.429382] workqueue events_power_efficient: flags=0x80 [ 5411.431635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5411.435046] pending: neigh_periodic_work, do_cache_clean [ 5411.437334] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5411.448333] pending: neigh_periodic_work, check_lifetime [ 5411.461153] workqueue mm_percpu_wq: flags=0x8 [ 5411.467245] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5411.473963] pending: vmstat_update [ 5411.480646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5411.489021] pending: vmstat_update [ 5411.490336] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5411.494174] pending: vmstat_update [ 5411.496362] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5411.503291] pending: vmstat_update [ 5411.507219] workqueue writeback: flags=0x4a [ 5411.511164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5411.517450] pending: wb_workfn, wb_workfn [ 5411.526347] workqueue xprtiod: flags=0xa [ 5411.531038] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5411.537292] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5411.548770] workqueue ipv6_addrconf: flags=0x40008 [ 5411.554078] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5411.561035] pending: addrconf_verify_work [ 5411.569633] workqueue knbd0-recv: flags=0x1a [ 5411.575236] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5411.589178] in-flight: 647:recv_work [ 5411.592216] workqueue ptlrpc_pinger: flags=0x2 [ 5411.595616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5411.599135] pending: ptlrpc_pinger_main [ptlrpc] [ 5411.608149] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5401s workers=2 idle: 652 [ 5412.059235] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5412.069999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5412.115295] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5412.137297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5412.147861] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5412.162850] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5412.184427] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5412.202102] RAX: 000000000000180f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5412.213671] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5412.239074] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5412.253885] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5412.266086] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5412.275494] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5412.280877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5412.288378] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5412.294709] Call Trace: [ 5412.295669] [ 5412.296532] ? show_regs.cold.9+0x22/0x2f [ 5412.298659] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5412.301347] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5412.303796] ? watchdog+0x40/0x40 [ 5412.304941] ? hrtimer_interrupt+0x128/0x2c0 [ 5412.306820] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5412.308406] ? apic_timer_interrupt+0xf/0x20 [ 5412.309857] [ 5412.310700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5412.312293] do_raw_spin_lock+0x177/0x190 [ 5412.313780] _raw_spin_lock+0x21/0x30 [ 5412.317943] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5412.319850] ? do_raw_spin_unlock+0x75/0x190 [ 5412.321043] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5412.327055] ? do_wait_intr+0xf0/0xf0 [ 5412.328949] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5412.341569] mdc_close+0x53b/0xd70 [mdc] [ 5412.348189] lmv_close+0x18d/0x490 [lmv] [ 5412.352774] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5412.357889] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5412.360883] ll_md_close+0x274/0x800 [lustre] [ 5412.364011] ll_file_release+0x128/0xc50 [lustre] [ 5412.367930] __fput+0x10d/0x440 [ 5412.370415] ____fput+0x12/0x20 [ 5412.374702] task_work_run+0xc6/0x110 [ 5412.378778] exit_to_usermode_loop+0x1dd/0x1f0 [ 5412.384962] do_syscall_64+0x3d6/0x440 [ 5412.388379] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5412.391877] RIP: 0033:0x7f1a63983cd8 [ 5412.395092] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5412.419594] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5412.427225] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5412.433493] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5412.446203] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5412.454644] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5412.462172] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5432.058328] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5432.062533] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5432.068104] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 5432.092277] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5432.092298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5432.092301] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5432.092316] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5432.092320] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5432.092326] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5432.092328] RDX: 0000000000002519 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5432.092330] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5432.092333] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5432.092335] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5432.092339] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5432.092341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5432.092344] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5432.092352] Call Trace: [ 5432.092358] [ 5432.092364] ? show_regs.cold.9+0x22/0x2f [ 5432.106688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5432.106764] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5432.106771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5432.106774] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5432.106788] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5432.106793] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5432.106798] RAX: 0000000000005554 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5432.106800] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5432.106803] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5432.106805] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5432.106807] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5432.106811] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5432.106813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5432.106815] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5432.106825] Call Trace: [ 5432.106830] [ 5432.106836] ? show_regs.cold.9+0x22/0x2f [ 5432.106846] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5432.106853] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5432.106857] ? watchdog+0x40/0x40 [ 5432.106861] ? hrtimer_interrupt+0x128/0x2c0 [ 5432.106864] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5432.106873] ? apic_timer_interrupt+0xf/0x20 [ 5432.106877] [ 5432.106881] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5432.106887] do_raw_spin_lock+0x177/0x190 [ 5432.106892] _raw_spin_lock+0x21/0x30 [ 5432.106897] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5432.107583] ? do_raw_spin_unlock+0x75/0x190 [ 5432.107590] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5432.107792] ? do_wait_intr+0xf0/0xf0 [ 5432.107800] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5432.108077] mdc_close+0x53b/0xd70 [mdc] [ 5432.108117] lmv_close+0x18d/0x490 [lmv] [ 5432.108150] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5432.108439] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5432.108524] ll_md_close+0x274/0x800 [lustre] [ 5432.108609] ll_file_release+0x128/0xc50 [lustre] [ 5432.108691] __fput+0x10d/0x440 [ 5432.108701] ____fput+0x12/0x20 [ 5432.108705] task_work_run+0xc6/0x110 [ 5432.108713] exit_to_usermode_loop+0x1dd/0x1f0 [ 5432.108720] do_syscall_64+0x3d6/0x440 [ 5432.108724] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5432.108732] RIP: 0033:0x7f1a63983cd8 [ 5432.108738] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5432.108742] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5432.108747] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5432.108750] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5432.108752] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5432.108754] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5432.108757] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5432.434376] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5432.436707] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5432.438900] ? watchdog+0x40/0x40 [ 5432.440206] ? hrtimer_interrupt+0x128/0x2c0 [ 5432.442790] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5432.445099] ? apic_timer_interrupt+0xf/0x20 [ 5432.448146] [ 5432.449223] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5432.451995] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5432.456769] do_raw_spin_lock+0x177/0x190 [ 5432.458928] _raw_spin_lock+0x21/0x30 [ 5432.460904] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5432.463842] obd_mod_free+0x46/0x140 [ptlrpc] [ 5432.466129] mdc_commit_open+0x91/0xb0 [mdc] [ 5432.468456] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5432.471769] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5432.474546] after_reply+0xe2c/0x1780 [ptlrpc] [ 5432.477635] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5432.481340] ? __next_timer_interrupt+0x160/0x160 [ 5432.483695] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5432.485969] ? do_wait_intr+0xf0/0xf0 [ 5432.487624] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5432.490122] kthread+0x1d1/0x200 [ 5432.491976] ? set_kthread_struct+0x70/0x70 [ 5432.493856] ret_from_fork+0x1f/0x30 [ 5436.053196] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5436.056461] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5436.078068] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5436.082997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5436.087788] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5436.091101] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5436.104186] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5436.108961] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5436.112467] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5436.117470] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5436.120503] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5436.124979] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5436.128357] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5436.132803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5436.135763] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5436.142969] Call Trace: [ 5436.143848] [ 5436.144672] ? show_regs.cold.9+0x22/0x2f [ 5436.147050] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5436.149406] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5436.150817] ? watchdog+0x40/0x40 [ 5436.153609] ? hrtimer_interrupt+0x128/0x2c0 [ 5436.155750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5436.158997] ? apic_timer_interrupt+0xf/0x20 [ 5436.163771] [ 5436.164694] ? native_safe_halt+0x2e/0x50 [ 5436.165734] kvm_wait+0xcd/0xe0 [ 5436.167498] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5436.171051] do_raw_spin_lock+0x177/0x190 [ 5436.174014] _raw_spin_lock+0x21/0x30 [ 5436.176679] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5436.179723] ? do_raw_spin_unlock+0x75/0x190 [ 5436.182175] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5436.185472] ? do_wait_intr+0xf0/0xf0 [ 5436.188520] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5436.192570] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5436.196945] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5436.201619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5436.205175] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5436.207745] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5436.268458] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5436.275469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5436.282592] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5436.297645] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5436.303144] lmv_intent_open+0x338/0x1150 [lmv] [ 5436.304785] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5436.308752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5436.320071] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5436.322155] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5436.324655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5436.327233] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5436.330102] ? do_raw_spin_unlock+0x75/0x190 [ 5436.332768] ? _raw_spin_unlock+0x12/0x30 [ 5436.334406] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5436.337438] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5436.340047] ll_atomic_open+0x36a/0xd70 [lustre] [ 5436.342682] lookup_open+0x356/0xb40 [ 5436.344836] do_last+0x313/0xfc0 [ 5436.346274] ? nd_jump_root+0xe5/0x160 [ 5436.348465] ? path_init+0x437/0x520 [ 5436.350686] path_openat+0xf7/0x500 [ 5436.351962] ? do_raw_spin_unlock+0x75/0x190 [ 5436.353992] do_filp_open+0x99/0x140 [ 5436.356080] ? getname_flags+0x6e/0x330 [ 5436.357768] ? __check_object_size+0xff/0x256 [ 5436.360138] ? do_raw_spin_unlock+0x75/0x190 [ 5436.362746] ? _raw_spin_unlock+0x12/0x30 [ 5436.365017] do_sys_openat2+0x2b4/0x410 [ 5436.366347] do_sys_open+0x73/0xa0 [ 5436.367918] __x64_sys_openat+0x24/0x30 [ 5436.369383] do_syscall_64+0xc1/0x440 [ 5436.371750] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5436.374561] RIP: 0033:0x7f1a63983332 [ 5436.376680] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5436.386154] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5436.390126] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5436.400661] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5436.406701] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5436.413125] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5436.417484] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5440.060418] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5440.069427] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5440.119994] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5440.135840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5440.144942] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5440.148589] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5440.161745] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5440.165714] RAX: 000000000000704e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5440.171987] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5440.175693] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5440.179012] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5440.183816] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5440.186669] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5440.191128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5440.193627] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5440.196761] Call Trace: [ 5440.197878] [ 5440.198676] ? show_regs.cold.9+0x22/0x2f [ 5440.200320] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5440.202652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5440.204709] ? watchdog+0x40/0x40 [ 5440.206403] ? hrtimer_interrupt+0x128/0x2c0 [ 5440.208910] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5440.211332] ? apic_timer_interrupt+0xf/0x20 [ 5440.213584] [ 5440.214120] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5440.216799] do_raw_spin_lock+0x177/0x190 [ 5440.220009] _raw_spin_lock+0x21/0x30 [ 5440.222357] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5440.229295] ? do_raw_spin_unlock+0x75/0x190 [ 5440.232438] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5440.237805] ? do_wait_intr+0xf0/0xf0 [ 5440.242723] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5440.245926] mdc_close+0x53b/0xd70 [mdc] [ 5440.247938] lmv_close+0x18d/0x490 [lmv] [ 5440.249067] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5440.252963] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5440.257816] ll_md_close+0x274/0x800 [lustre] [ 5440.260593] ll_file_release+0x128/0xc50 [lustre] [ 5440.262560] __fput+0x10d/0x440 [ 5440.264438] ____fput+0x12/0x20 [ 5440.265289] task_work_run+0xc6/0x110 [ 5440.266967] exit_to_usermode_loop+0x1dd/0x1f0 [ 5440.268205] do_syscall_64+0x3d6/0x440 [ 5440.269268] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5440.271977] RIP: 0033:0x7f1a63983cd8 [ 5440.273920] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5440.282650] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5440.285589] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5440.288310] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5440.292399] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5440.295758] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5440.300373] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5442.022059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3419s! [ 5442.026046] Showing busy workqueues and worker pools: [ 5442.031436] workqueue events: flags=0x0 [ 5442.037173] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5442.050715] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5442.053634] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5442.059315] pending: sec_gc_main [ptlrpc] [ 5442.063762] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5442.067920] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5442.074629] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5442.079957] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5442.089497] workqueue events_long: flags=0x0 [ 5442.091731] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5442.094747] pending: lu_objects_flush [obdclass] [ 5442.097505] workqueue events_unbound: flags=0x2 [ 5442.100374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5442.104211] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5442.109910] workqueue events_power_efficient: flags=0x80 [ 5442.112981] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5442.116685] pending: neigh_periodic_work, do_cache_clean [ 5442.120656] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5442.129301] pending: neigh_periodic_work, check_lifetime [ 5442.132788] workqueue mm_percpu_wq: flags=0x8 [ 5442.135976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5442.142080] pending: vmstat_update [ 5442.146047] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5442.152717] pending: vmstat_update [ 5442.155661] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5442.159591] pending: vmstat_update [ 5442.162857] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5442.170318] pending: vmstat_update [ 5442.172632] workqueue writeback: flags=0x4a [ 5442.174743] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5442.186666] pending: wb_workfn, wb_workfn [ 5442.188469] workqueue xprtiod: flags=0xa [ 5442.192177] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5442.196816] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5442.202748] workqueue ipv6_addrconf: flags=0x40008 [ 5442.214260] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5442.223383] pending: addrconf_verify_work [ 5442.228124] workqueue knbd0-recv: flags=0x1a [ 5442.231116] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5442.234926] in-flight: 647:recv_work [ 5442.239927] workqueue ptlrpc_pinger: flags=0x2 [ 5442.242901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5442.248130] pending: ptlrpc_pinger_main [ptlrpc] [ 5442.254289] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5432s workers=2 idle: 652 [ 5460.057034] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 5460.060157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5460.062385] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 5460.083596] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5460.083611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5460.083614] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5460.083628] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5460.083633] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5460.083639] RAX: 0000000000007d93 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5460.083642] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5460.083644] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5460.083646] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5460.083649] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5460.083652] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5460.083655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5460.083658] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5460.083666] Call Trace: [ 5460.083671] [ 5460.083677] ? show_regs.cold.9+0x22/0x2f [ 5460.083686] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5460.083692] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5460.083697] ? watchdog+0x40/0x40 [ 5460.083702] ? hrtimer_interrupt+0x128/0x2c0 [ 5460.083706] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5460.083714] ? apic_timer_interrupt+0xf/0x20 [ 5460.083719] [ 5460.083722] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5460.083729] do_raw_spin_lock+0x177/0x190 [ 5460.083733] _raw_spin_lock+0x21/0x30 [ 5460.083738] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5460.093494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5460.093912] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5460.093919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5460.093922] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5460.093935] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5460.093939] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5460.093943] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5460.093946] RDX: 00000000000051ff RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5460.093948] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5460.093950] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5460.093952] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5460.093954] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5460.093956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5460.093958] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5460.093981] Call Trace: [ 5460.093987] [ 5460.093993] ? show_regs.cold.9+0x22/0x2f [ 5460.094001] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5460.094007] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5460.094011] ? watchdog+0x40/0x40 [ 5460.094016] ? hrtimer_interrupt+0x128/0x2c0 [ 5460.094020] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5460.094028] ? apic_timer_interrupt+0xf/0x20 [ 5460.094033] [ 5460.094036] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5460.094041] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5460.094047] do_raw_spin_lock+0x177/0x190 [ 5460.094051] _raw_spin_lock+0x21/0x30 [ 5460.094057] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5460.094316] obd_mod_free+0x46/0x140 [ptlrpc] [ 5460.103773] ? do_raw_spin_unlock+0x75/0x190 [ 5460.103795] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5460.104145] ? do_wait_intr+0xf0/0xf0 [ 5460.104154] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5460.104709] mdc_close+0x53b/0xd70 [mdc] [ 5460.104748] lmv_close+0x18d/0x490 [lmv] [ 5460.104779] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5460.104897] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5460.105063] ll_md_close+0x274/0x800 [lustre] [ 5460.105159] ll_file_release+0x128/0xc50 [lustre] [ 5460.105365] __fput+0x10d/0x440 [ 5460.105374] ____fput+0x12/0x20 [ 5460.105379] task_work_run+0xc6/0x110 [ 5460.105509] exit_to_usermode_loop+0x1dd/0x1f0 [ 5460.105519] do_syscall_64+0x3d6/0x440 [ 5460.105524] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5460.105533] RIP: 0033:0x7f1a63983cd8 [ 5460.105540] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5460.105544] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5460.105549] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5460.105552] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5460.105554] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5460.105557] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5460.105559] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5460.541269] mdc_commit_open+0x91/0xb0 [mdc] [ 5460.545455] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5460.548182] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5460.552750] after_reply+0xe2c/0x1780 [ptlrpc] [ 5460.553980] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5460.556406] ? __next_timer_interrupt+0x160/0x160 [ 5460.558412] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5460.560370] ? do_wait_intr+0xf0/0xf0 [ 5460.562129] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5460.564901] kthread+0x1d1/0x200 [ 5460.566351] ? set_kthread_struct+0x70/0x70 [ 5460.571521] ret_from_fork+0x1f/0x30 [ 5468.053062] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5468.056250] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5468.057237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [ 5468.062990] Modules linked in: lustre(O) [ 5468.069909] osc(O) [ 5468.069918] osp(O) [ 5468.072151] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5468.072208] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5468.072216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5468.072218] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5468.073264] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5468.073403] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5468.073410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5468.073412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5468.073424] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5468.073428] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5468.073433] RAX: 0000000000000c19 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5468.073436] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5468.073438] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5468.073440] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5468.073442] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5468.073444] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5468.073447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5468.073454] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5468.073466] Call Trace: [ 5468.073473] [ 5468.073478] ? show_regs.cold.9+0x22/0x2f [ 5468.073488] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5468.073494] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5468.073498] ? watchdog+0x40/0x40 [ 5468.073502] ? hrtimer_interrupt+0x128/0x2c0 [ 5468.073506] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5468.073515] ? apic_timer_interrupt+0xf/0x20 [ 5468.073519] [ 5468.073522] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5468.073528] do_raw_spin_lock+0x177/0x190 [ 5468.073548] _raw_spin_lock+0x21/0x30 [ 5468.073555] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5468.075156] ? do_raw_spin_unlock+0x75/0x190 [ 5468.075166] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5468.075577] ? do_wait_intr+0xf0/0xf0 [ 5468.075586] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5468.075808] mdc_close+0x53b/0xd70 [mdc] [ 5468.075862] lmv_close+0x18d/0x490 [lmv] [ 5468.075907] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5468.076548] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5468.076629] ll_md_close+0x274/0x800 [lustre] [ 5468.076711] ll_file_release+0x128/0xc50 [lustre] [ 5468.076790] __fput+0x10d/0x440 [ 5468.076799] ____fput+0x12/0x20 [ 5468.076803] task_work_run+0xc6/0x110 [ 5468.076810] exit_to_usermode_loop+0x1dd/0x1f0 [ 5468.076817] do_syscall_64+0x3d6/0x440 [ 5468.076821] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5468.076830] RIP: 0033:0x7f1a63983cd8 [ 5468.076838] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5468.076842] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5468.076846] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5468.076848] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5468.076850] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5468.076852] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5468.076854] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5468.371376] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5468.388185] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5468.393496] RAX: 0000000000006486 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5468.396673] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5468.400677] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5468.405946] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5468.413214] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5468.417517] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5468.425562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5468.429344] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5468.436446] Call Trace: [ 5468.439312] [ 5468.442137] ? show_regs.cold.9+0x22/0x2f [ 5468.444706] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5468.447387] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5468.451609] ? watchdog+0x40/0x40 [ 5468.452629] ? hrtimer_interrupt+0x128/0x2c0 [ 5468.456913] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5468.459460] ? apic_timer_interrupt+0xf/0x20 [ 5468.462783] [ 5468.463603] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5468.466138] do_raw_spin_lock+0x177/0x190 [ 5468.468671] _raw_spin_lock+0x21/0x30 [ 5468.477054] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5468.478925] ? do_raw_spin_unlock+0x75/0x190 [ 5468.488184] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5468.498329] ? do_wait_intr+0xf0/0xf0 [ 5468.506149] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5468.514397] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5468.520180] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5468.527959] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5468.530250] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5468.535334] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5468.538444] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5468.543444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5468.553257] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5468.558345] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5468.564715] lmv_intent_open+0x338/0x1150 [lmv] [ 5468.569690] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5468.575058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5468.579624] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5468.581950] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5468.584450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5468.587353] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5468.596487] ? do_raw_spin_unlock+0x75/0x190 [ 5468.599712] ? _raw_spin_unlock+0x12/0x30 [ 5468.603703] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5468.609281] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5468.611421] ll_atomic_open+0x36a/0xd70 [lustre] [ 5468.613638] lookup_open+0x356/0xb40 [ 5468.615701] do_last+0x313/0xfc0 [ 5468.619276] ? nd_jump_root+0xe5/0x160 [ 5468.621321] ? path_init+0x437/0x520 [ 5468.624268] path_openat+0xf7/0x500 [ 5468.626676] ? do_raw_spin_unlock+0x75/0x190 [ 5468.628715] do_filp_open+0x99/0x140 [ 5468.629862] ? getname_flags+0x6e/0x330 [ 5468.640523] ? __check_object_size+0xff/0x256 [ 5468.644696] ? do_raw_spin_unlock+0x75/0x190 [ 5468.649256] ? _raw_spin_unlock+0x12/0x30 [ 5468.650971] do_sys_openat2+0x2b4/0x410 [ 5468.652574] do_sys_open+0x73/0xa0 [ 5468.661298] __x64_sys_openat+0x24/0x30 [ 5468.664763] do_syscall_64+0xc1/0x440 [ 5468.666610] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5468.671336] RIP: 0033:0x7f1a63983332 [ 5468.673243] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5468.698626] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5468.700682] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5468.710171] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5468.719667] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5468.726427] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5468.729303] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5472.739805] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3449s! [ 5472.743563] Showing busy workqueues and worker pools: [ 5472.745791] workqueue events: flags=0x0 [ 5472.747809] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5472.751005] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5472.761924] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5472.770252] pending: sec_gc_main [ptlrpc] [ 5472.774297] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5472.778189] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5472.788018] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5472.795241] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5472.802159] workqueue events_long: flags=0x0 [ 5472.804043] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5472.811796] pending: lu_objects_flush [obdclass] [ 5472.813627] workqueue events_unbound: flags=0x2 [ 5472.819772] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5472.828270] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5472.843916] workqueue events_power_efficient: flags=0x80 [ 5472.848517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5472.856375] pending: neigh_periodic_work, do_cache_clean [ 5472.861332] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5472.868589] pending: neigh_periodic_work, check_lifetime [ 5472.874167] workqueue mm_percpu_wq: flags=0x8 [ 5472.877646] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5472.884815] pending: vmstat_update [ 5472.888134] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5472.895485] pending: vmstat_update [ 5472.898362] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5472.904592] pending: vmstat_update [ 5472.911008] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5472.916474] pending: vmstat_update [ 5472.918519] workqueue writeback: flags=0x4a [ 5472.924079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5472.930270] pending: wb_workfn, wb_workfn [ 5472.932625] workqueue xprtiod: flags=0xa [ 5472.936232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5472.939991] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5472.948679] workqueue ipv6_addrconf: flags=0x40008 [ 5472.954622] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5472.958378] pending: addrconf_verify_work [ 5472.961030] workqueue knbd0-recv: flags=0x1a [ 5472.963322] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5472.966595] in-flight: 647:recv_work [ 5472.968612] workqueue ptlrpc_pinger: flags=0x2 [ 5472.970979] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5472.975236] pending: ptlrpc_pinger_main [ptlrpc] [ 5472.980390] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5463s workers=2 idle: 652 [ 5488.058741] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 5488.061946] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 5488.068492] Modules linked in: [ 5488.076541] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 5488.078471] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5488.078680] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5488.078704] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5488.078708] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5488.095488] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5488.095532] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5488.095540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5488.095542] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5488.095560] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5488.095564] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5488.095569] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5488.095572] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5488.095574] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5488.095576] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5488.095579] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5488.095581] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5488.095584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5488.095587] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5488.095596] Call Trace: [ 5488.095606] [ 5488.095613] ? show_regs.cold.9+0x22/0x2f [ 5488.095625] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5488.150002] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5488.150015] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5488.150023] RAX: 0000000000000d73 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5488.150030] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5488.150033] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5488.150035] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5488.150038] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5488.150041] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5488.150045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5488.150047] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5488.150055] Call Trace: [ 5488.150060] [ 5488.150065] ? show_regs.cold.9+0x22/0x2f [ 5488.150078] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5488.150084] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5488.150088] ? watchdog+0x40/0x40 [ 5488.150092] ? hrtimer_interrupt+0x128/0x2c0 [ 5488.150096] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5488.150103] ? apic_timer_interrupt+0xf/0x20 [ 5488.150399] [ 5488.150403] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5488.150412] do_raw_spin_lock+0x177/0x190 [ 5488.150418] _raw_spin_lock+0x21/0x30 [ 5488.150423] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5488.150964] ? do_raw_spin_unlock+0x75/0x190 [ 5488.150970] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5488.151217] ? do_wait_intr+0xf0/0xf0 [ 5488.151224] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5488.151465] mdc_close+0x53b/0xd70 [mdc] [ 5488.151508] lmv_close+0x18d/0x490 [lmv] [ 5488.151538] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5488.151667] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5488.152043] ll_md_close+0x274/0x800 [lustre] [ 5488.152127] ll_file_release+0x128/0xc50 [lustre] [ 5488.152260] __fput+0x10d/0x440 [ 5488.152271] ____fput+0x12/0x20 [ 5488.152275] task_work_run+0xc6/0x110 [ 5488.152282] exit_to_usermode_loop+0x1dd/0x1f0 [ 5488.152288] do_syscall_64+0x3d6/0x440 [ 5488.152292] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5488.152299] RIP: 0033:0x7f1a63983cd8 [ 5488.152306] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5488.152310] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5488.152315] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5488.152318] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5488.152320] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5488.152323] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5488.152325] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5488.620839] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5488.626319] ? watchdog+0x40/0x40 [ 5488.634197] ? hrtimer_interrupt+0x128/0x2c0 [ 5488.638296] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5488.645114] ? apic_timer_interrupt+0xf/0x20 [ 5488.652801] [ 5488.654032] ? native_safe_halt+0x2e/0x50 [ 5488.657463] kvm_wait+0xcd/0xe0 [ 5488.660170] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5488.664184] do_raw_spin_lock+0x177/0x190 [ 5488.666847] _raw_spin_lock+0x21/0x30 [ 5488.668855] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5488.672337] obd_mod_free+0x46/0x140 [ptlrpc] [ 5488.675478] mdc_commit_open+0x91/0xb0 [mdc] [ 5488.677821] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5488.681352] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5488.684898] after_reply+0xe2c/0x1780 [ptlrpc] [ 5488.687126] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5488.690253] ? __next_timer_interrupt+0x160/0x160 [ 5488.692876] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5488.694647] ? do_wait_intr+0xf0/0xf0 [ 5488.696245] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5488.698259] kthread+0x1d1/0x200 [ 5488.699176] ? set_kthread_struct+0x70/0x70 [ 5488.701316] ret_from_fork+0x1f/0x30 [ 5496.070983] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5496.089302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5496.191174] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5496.207286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5496.239251] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5496.244186] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5496.255451] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5496.272633] RAX: 000000000000489a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5496.283277] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5496.291256] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5496.293529] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5496.297730] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5496.308286] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5496.318042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5496.333213] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5496.350280] Call Trace: [ 5496.352768] [ 5496.355193] ? show_regs.cold.9+0x22/0x2f [ 5496.362939] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5496.375911] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5496.382353] ? watchdog+0x40/0x40 [ 5496.383333] ? hrtimer_interrupt+0x128/0x2c0 [ 5496.390377] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5496.404030] ? apic_timer_interrupt+0xf/0x20 [ 5496.411229] [ 5496.415102] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5496.422026] do_raw_spin_lock+0x177/0x190 [ 5496.423486] _raw_spin_lock+0x21/0x30 [ 5496.434036] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5496.439588] ? do_raw_spin_unlock+0x75/0x190 [ 5496.442110] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5496.450755] ? do_wait_intr+0xf0/0xf0 [ 5496.455716] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5496.459915] mdc_close+0x53b/0xd70 [mdc] [ 5496.461105] lmv_close+0x18d/0x490 [lmv] [ 5496.465108] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5496.468207] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5496.471836] ll_md_close+0x274/0x800 [lustre] [ 5496.474064] ll_file_release+0x128/0xc50 [lustre] [ 5496.477161] __fput+0x10d/0x440 [ 5496.478748] ____fput+0x12/0x20 [ 5496.480566] task_work_run+0xc6/0x110 [ 5496.482485] exit_to_usermode_loop+0x1dd/0x1f0 [ 5496.485051] do_syscall_64+0x3d6/0x440 [ 5496.486987] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5496.489720] RIP: 0033:0x7f1a63983cd8 [ 5496.491943] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5496.500974] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5496.505985] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5496.510109] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5496.513432] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5496.517361] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5496.522059] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5500.053085] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 5500.056186] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5500.078931] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5500.084012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5500.089049] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5500.091474] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5500.104081] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5500.111117] RAX: 0000000000007e8d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5500.115041] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5500.121098] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5500.124138] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5500.127774] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5500.132059] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5500.136178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5500.139791] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5500.143404] Call Trace: [ 5500.144935] [ 5500.146158] ? show_regs.cold.9+0x22/0x2f [ 5500.148340] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5500.150945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5500.153283] ? watchdog+0x40/0x40 [ 5500.155121] ? hrtimer_interrupt+0x128/0x2c0 [ 5500.157409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5500.160120] ? apic_timer_interrupt+0xf/0x20 [ 5500.162101] [ 5500.163362] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5500.166223] do_raw_spin_lock+0x177/0x190 [ 5500.168283] _raw_spin_lock+0x21/0x30 [ 5500.170310] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5500.173428] ? do_raw_spin_unlock+0x75/0x190 [ 5500.175373] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5500.178173] ? do_wait_intr+0xf0/0xf0 [ 5500.179412] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5500.182055] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5500.185003] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5500.188216] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5500.190893] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5500.193866] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5500.196092] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5500.198309] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5500.201152] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5500.204390] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5500.207434] lmv_intent_open+0x338/0x1150 [lmv] [ 5500.210170] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5500.212950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5500.215425] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5500.218139] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5500.220410] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5500.223552] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5500.225904] ? do_raw_spin_unlock+0x75/0x190 [ 5500.228017] ? _raw_spin_unlock+0x12/0x30 [ 5500.230168] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5500.233239] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5500.235952] ll_atomic_open+0x36a/0xd70 [lustre] [ 5500.238537] lookup_open+0x356/0xb40 [ 5500.240912] do_last+0x313/0xfc0 [ 5500.242387] ? nd_jump_root+0xe5/0x160 [ 5500.244233] ? path_init+0x437/0x520 [ 5500.246229] path_openat+0xf7/0x500 [ 5500.248108] ? do_raw_spin_unlock+0x75/0x190 [ 5500.250387] do_filp_open+0x99/0x140 [ 5500.252062] ? getname_flags+0x6e/0x330 [ 5500.254233] ? __check_object_size+0xff/0x256 [ 5500.256299] ? do_raw_spin_unlock+0x75/0x190 [ 5500.258310] ? _raw_spin_unlock+0x12/0x30 [ 5500.267489] do_sys_openat2+0x2b4/0x410 [ 5500.269396] do_sys_open+0x73/0xa0 [ 5500.271852] __x64_sys_openat+0x24/0x30 [ 5500.273772] do_syscall_64+0xc1/0x440 [ 5500.275734] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5500.277895] RIP: 0033:0x7f1a63983332 [ 5500.279832] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5500.291753] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5500.296253] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5500.300503] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5500.305423] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5500.309451] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5500.313703] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5502.485758] rcu: INFO: rcu_sched self-detected stall on CPU [ 5502.491139] rcu: 3-...!: (2114637 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 5502.501666] (t=3480092 jiffies g=269633 q=1355) [ 5502.504441] rcu: rcu_sched kthread starved for 3479738 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5502.512436] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5502.516703] rcu: RCU grace-period kthread stack dump: [ 5502.520308] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5502.525157] Call Trace: [ 5502.526248] __schedule+0x351/0xcb0 [ 5502.527745] schedule+0xc0/0x180 [ 5502.529251] schedule_timeout+0xb4/0x190 [ 5502.530990] ? do_raw_spin_unlock+0x75/0x190 [ 5502.532824] ? __next_timer_interrupt+0x160/0x160 [ 5502.534947] ? prepare_to_swait_event+0xf3/0x140 [ 5502.536914] rcu_gp_kthread+0x8dd/0x15b0 [ 5502.538715] ? force_qs_rnp+0x2c0/0x2c0 [ 5502.540381] kthread+0x1d1/0x200 [ 5502.543532] ? set_kthread_struct+0x70/0x70 [ 5502.545175] ret_from_fork+0x1f/0x30 [ 5502.546810] rcu: Stack dump where RCU GP kthread last ran: [ 5502.549281] Sending NMI from CPU 3 to CPUs 2: [ 5502.551182] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 5502.554417] NMI backtrace for cpu 3 [ 5502.556186] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5502.561224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5502.565272] Call Trace: [ 5502.566307] [ 5502.567187] ? dump_stack+0xbb/0x10e [ 5502.568929] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5502.571201] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5502.573508] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5502.575747] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5502.578171] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5502.580790] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5502.582743] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5502.585223] ? update_process_times+0x6d/0xc0 [ 5502.587188] ? tick_sched_handle.isra.18+0x27/0x90 [ 5502.589370] ? tick_sched_timer+0x53/0xb0 [ 5502.591900] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5502.593993] ? tick_sched_handle.isra.18+0x90/0x90 [ 5502.595988] ? hrtimer_interrupt+0x128/0x2c0 [ 5502.597120] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5502.598423] ? apic_timer_interrupt+0xf/0x20 [ 5502.599510] [ 5502.600811] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5502.603203] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5502.605175] ? do_raw_spin_lock+0x177/0x190 [ 5502.606534] ? _raw_spin_lock+0x21/0x30 [ 5502.607664] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5502.609657] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5502.611676] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5502.613408] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5502.615402] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5502.617379] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5502.618890] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5502.620636] ? __next_timer_interrupt+0x160/0x160 [ 5502.621867] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5502.623479] ? do_wait_intr+0xf0/0xf0 [ 5502.624368] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5502.625955] ? kthread+0x1d1/0x200 [ 5502.626911] ? set_kthread_struct+0x70/0x70 [ 5502.628092] ? ret_from_fork+0x1f/0x30 [ 5503.456437] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3480s! [ 5503.461857] Showing busy workqueues and worker pools: [ 5503.465290] workqueue events: flags=0x0 [ 5503.467460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5503.470731] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5503.474145] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5503.477028] pending: sec_gc_main [ptlrpc] [ 5503.479667] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5503.483374] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5503.488247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5503.491583] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5503.496026] workqueue events_long: flags=0x0 [ 5503.498218] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5503.501070] pending: lu_objects_flush [obdclass] [ 5503.503476] workqueue events_unbound: flags=0x2 [ 5503.505662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5503.509028] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5503.513249] workqueue events_power_efficient: flags=0x80 [ 5503.515846] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5503.519060] pending: neigh_periodic_work, do_cache_clean [ 5503.521445] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5503.525462] pending: neigh_periodic_work, check_lifetime [ 5503.528134] workqueue mm_percpu_wq: flags=0x8 [ 5503.530483] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5503.532981] pending: vmstat_update [ 5503.535275] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5503.538515] pending: vmstat_update [ 5503.539871] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5503.542307] pending: vmstat_update [ 5503.543943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5503.548093] pending: vmstat_update [ 5503.549492] workqueue writeback: flags=0x4a [ 5503.553012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5503.555726] pending: wb_workfn, wb_workfn [ 5503.557976] workqueue xprtiod: flags=0xa [ 5503.559750] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5503.562520] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5503.565949] workqueue ipv6_addrconf: flags=0x40008 [ 5503.567853] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5503.570485] pending: addrconf_verify_work [ 5503.572430] workqueue knbd0-recv: flags=0x1a [ 5503.574216] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5503.576872] in-flight: 647:recv_work [ 5503.578485] workqueue ptlrpc_pinger: flags=0x2 [ 5503.580204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5503.582702] pending: ptlrpc_pinger_main [ptlrpc] [ 5503.585679] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5493s workers=2 idle: 652 [ 5524.056099] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5524.060630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5524.077107] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5524.080762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5524.084007] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5524.086049] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5524.099934] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5524.103884] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5524.108489] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5524.112124] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5524.116693] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5524.122118] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5524.128624] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5524.137486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5524.146951] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5524.153913] Call Trace: [ 5524.158747] [ 5524.162718] ? show_regs.cold.9+0x22/0x2f [ 5524.167948] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5524.175412] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5524.178518] ? watchdog+0x40/0x40 [ 5524.180067] ? hrtimer_interrupt+0x128/0x2c0 [ 5524.183949] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5524.189830] ? apic_timer_interrupt+0xf/0x20 [ 5524.195970] [ 5524.197145] ? native_safe_halt+0x2e/0x50 [ 5524.202571] kvm_wait+0xcd/0xe0 [ 5524.205117] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5524.214536] do_raw_spin_lock+0x177/0x190 [ 5524.220427] _raw_spin_lock+0x21/0x30 [ 5524.224006] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5524.226976] ? do_raw_spin_unlock+0x75/0x190 [ 5524.229122] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5524.231698] ? do_wait_intr+0xf0/0xf0 [ 5524.234178] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5524.236925] mdc_close+0x53b/0xd70 [mdc] [ 5524.241281] lmv_close+0x18d/0x490 [lmv] [ 5524.242345] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5524.245224] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5524.247624] ll_md_close+0x274/0x800 [lustre] [ 5524.249716] ll_file_release+0x128/0xc50 [lustre] [ 5524.251894] __fput+0x10d/0x440 [ 5524.253641] ____fput+0x12/0x20 [ 5524.254975] task_work_run+0xc6/0x110 [ 5524.256729] exit_to_usermode_loop+0x1dd/0x1f0 [ 5524.258736] do_syscall_64+0x3d6/0x440 [ 5524.260555] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5524.262824] RIP: 0033:0x7f1a63983cd8 [ 5524.264786] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5524.273560] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5524.277064] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5524.280887] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5524.284610] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5524.288646] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5524.291985] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5528.053071] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5528.057030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5528.058206] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5528.058215] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5528.058299] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5528.058304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5528.058307] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5528.058334] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5528.058339] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5528.058344] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5528.058347] RDX: 00000000000022e6 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5528.058350] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5528.058352] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5528.058354] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5528.058358] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5528.058360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5528.058363] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5528.058372] Call Trace: [ 5528.058378] [ 5528.058383] ? show_regs.cold.9+0x22/0x2f [ 5528.058394] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5528.058401] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5528.058405] ? watchdog+0x40/0x40 [ 5528.058409] ? hrtimer_interrupt+0x128/0x2c0 [ 5528.058413] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5528.058423] ? apic_timer_interrupt+0xf/0x20 [ 5528.058428] [ 5528.058431] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5528.058435] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5528.058440] do_raw_spin_lock+0x177/0x190 [ 5528.058444] _raw_spin_lock+0x21/0x30 [ 5528.058450] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5528.058743] obd_mod_free+0x46/0x140 [ptlrpc] [ 5528.059175] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5528.059237] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5528.059242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5528.059245] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5528.059255] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5528.059258] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5528.059263] RAX: 0000000000006b9b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5528.059265] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5528.059268] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5528.059270] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5528.059272] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5528.059276] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5528.059279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5528.059281] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5528.059288] Call Trace: [ 5528.059291] [ 5528.059294] ? show_regs.cold.9+0x22/0x2f [ 5528.059301] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5528.059307] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5528.059320] ? watchdog+0x40/0x40 [ 5528.059325] ? hrtimer_interrupt+0x128/0x2c0 [ 5528.059328] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5528.059335] ? apic_timer_interrupt+0xf/0x20 [ 5528.063274] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5528.063414] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5528.063421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5528.063424] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5528.063438] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5528.063448] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5528.063452] RAX: 0000000000001711 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5528.063455] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5528.063457] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5528.063460] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5528.063463] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5528.063467] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5528.063470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5528.063472] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5528.063481] Call Trace: [ 5528.063487] [ 5528.063492] ? show_regs.cold.9+0x22/0x2f [ 5528.063503] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5528.063509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5528.063513] ? watchdog+0x40/0x40 [ 5528.063518] ? hrtimer_interrupt+0x128/0x2c0 [ 5528.063522] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5528.063530] ? apic_timer_interrupt+0xf/0x20 [ 5528.063541] [ 5528.063545] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5528.063551] do_raw_spin_lock+0x177/0x190 [ 5528.063557] _raw_spin_lock+0x21/0x30 [ 5528.063562] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5528.064349] ? do_raw_spin_unlock+0x75/0x190 [ 5528.064357] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5528.064617] ? do_wait_intr+0xf0/0xf0 [ 5528.064625] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5528.064827] mdc_close+0x53b/0xd70 [mdc] [ 5528.065235] mdc_commit_open+0x91/0xb0 [mdc] [ 5528.065268] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5528.065512] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5528.065708] after_reply+0xe2c/0x1780 [ptlrpc] [ 5528.065904] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5528.066463] ? __next_timer_interrupt+0x160/0x160 [ 5528.066474] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5528.066674] ? do_wait_intr+0xf0/0xf0 [ 5528.066684] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5528.066887] kthread+0x1d1/0x200 [ 5528.066893] ? set_kthread_struct+0x70/0x70 [ 5528.066897] ret_from_fork+0x1f/0x30 [ 5528.070874] [ 5528.070892] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5528.070907] do_raw_spin_lock+0x177/0x190 [ 5528.070913] _raw_spin_lock+0x21/0x30 [ 5528.070919] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5528.093271] lmv_close+0x18d/0x490 [lmv] [ 5528.093323] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5528.093441] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5528.093518] ll_md_close+0x274/0x800 [lustre] [ 5528.093598] ll_file_release+0x128/0xc50 [lustre] [ 5528.093942] __fput+0x10d/0x440 [ 5528.093951] ____fput+0x12/0x20 [ 5528.093956] task_work_run+0xc6/0x110 [ 5528.093963] exit_to_usermode_loop+0x1dd/0x1f0 [ 5528.093970] do_syscall_64+0x3d6/0x440 [ 5528.093975] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5528.093983] RIP: 0033:0x7f1a63983cd8 [ 5528.093990] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5528.093994] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5528.094000] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5528.094002] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5528.094004] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5528.094006] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5528.094008] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5528.734495] ? do_raw_spin_unlock+0x75/0x190 [ 5528.738668] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5528.744504] ? do_wait_intr+0xf0/0xf0 [ 5528.748493] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5528.757185] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5528.763714] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5528.771892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5528.775972] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5528.783107] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5528.784429] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5528.786138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5528.793571] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5528.796857] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5528.805278] lmv_intent_open+0x338/0x1150 [lmv] [ 5528.808551] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5528.814353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5528.818370] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5528.824814] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5528.828803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5528.832429] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5528.836965] ? do_raw_spin_unlock+0x75/0x190 [ 5528.838949] ? _raw_spin_unlock+0x12/0x30 [ 5528.841854] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5528.843985] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5528.847424] ll_atomic_open+0x36a/0xd70 [lustre] [ 5528.849895] lookup_open+0x356/0xb40 [ 5528.851047] do_last+0x313/0xfc0 [ 5528.853347] ? nd_jump_root+0xe5/0x160 [ 5528.854590] ? path_init+0x437/0x520 [ 5528.862838] path_openat+0xf7/0x500 [ 5528.872485] ? do_raw_spin_unlock+0x75/0x190 [ 5528.884587] do_filp_open+0x99/0x140 [ 5528.891173] ? getname_flags+0x6e/0x330 [ 5528.906352] ? __check_object_size+0xff/0x256 [ 5528.915918] ? do_raw_spin_unlock+0x75/0x190 [ 5528.925658] ? _raw_spin_unlock+0x12/0x30 [ 5528.931199] do_sys_openat2+0x2b4/0x410 [ 5528.935140] do_sys_open+0x73/0xa0 [ 5528.939411] __x64_sys_openat+0x24/0x30 [ 5528.947915] do_syscall_64+0xc1/0x440 [ 5528.949272] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5528.952947] RIP: 0033:0x7f1a63983332 [ 5528.955818] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5528.983210] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5528.992435] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5529.003668] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5529.011830] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5529.024564] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5529.037962] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5533.664032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3510s! [ 5533.666854] Showing busy workqueues and worker pools: [ 5533.668785] workqueue events: flags=0x0 [ 5533.670134] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5533.672320] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5533.674866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5533.677097] pending: sec_gc_main [ptlrpc] [ 5533.679778] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5533.681927] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5533.685912] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5533.687919] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5533.692350] workqueue events_long: flags=0x0 [ 5533.693507] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5533.696170] pending: lu_objects_flush [obdclass] [ 5533.697933] workqueue events_unbound: flags=0x2 [ 5533.699447] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5533.701597] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5533.707383] workqueue events_power_efficient: flags=0x80 [ 5533.713364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5533.718450] pending: neigh_periodic_work, do_cache_clean [ 5533.722084] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5533.728146] pending: neigh_periodic_work, check_lifetime [ 5533.731063] workqueue mm_percpu_wq: flags=0x8 [ 5533.737445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5533.744049] pending: vmstat_update [ 5533.748876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5533.751910] pending: vmstat_update [ 5533.753744] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5533.756335] pending: vmstat_update [ 5533.757618] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5533.761236] pending: vmstat_update [ 5533.763390] workqueue writeback: flags=0x4a [ 5533.764919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5533.767531] pending: wb_workfn, wb_workfn [ 5533.769508] workqueue xprtiod: flags=0xa [ 5533.771831] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5533.776077] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5533.780489] workqueue ipv6_addrconf: flags=0x40008 [ 5533.790125] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5533.799314] pending: addrconf_verify_work [ 5533.804711] workqueue knbd0-recv: flags=0x1a [ 5533.807866] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5533.811370] in-flight: 647:recv_work [ 5533.814392] workqueue ptlrpc_pinger: flags=0x2 [ 5533.820950] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5533.824068] pending: ptlrpc_pinger_main [ptlrpc] [ 5533.826822] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5523s workers=2 idle: 652 [ 5552.056010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5552.060748] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5552.080518] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5552.085596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5552.089484] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5552.092332] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 5552.100471] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5552.103728] RAX: 000000000000008d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5552.107842] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5552.110961] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5552.113979] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5552.117031] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5552.120173] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5552.123904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5552.126868] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5552.129743] Call Trace: [ 5552.130587] [ 5552.131283] ? show_regs.cold.9+0x22/0x2f [ 5552.132745] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5552.135758] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5552.138726] ? watchdog+0x40/0x40 [ 5552.140415] ? hrtimer_interrupt+0x128/0x2c0 [ 5552.142556] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5552.145665] ? apic_timer_interrupt+0xf/0x20 [ 5552.148301] [ 5552.149814] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5552.152712] do_raw_spin_lock+0x177/0x190 [ 5552.153973] _raw_spin_lock+0x21/0x30 [ 5552.155833] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5552.161280] ? do_raw_spin_unlock+0x75/0x190 [ 5552.162779] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5552.165675] ? do_wait_intr+0xf0/0xf0 [ 5552.167549] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5552.169909] mdc_close+0x53b/0xd70 [mdc] [ 5552.171750] lmv_close+0x18d/0x490 [lmv] [ 5552.173977] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5552.177194] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5552.179408] ll_md_close+0x274/0x800 [lustre] [ 5552.181656] ll_file_release+0x128/0xc50 [lustre] [ 5552.183323] __fput+0x10d/0x440 [ 5552.184643] ____fput+0x12/0x20 [ 5552.185960] task_work_run+0xc6/0x110 [ 5552.187594] exit_to_usermode_loop+0x1dd/0x1f0 [ 5552.189589] do_syscall_64+0x3d6/0x440 [ 5552.191185] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5552.193444] RIP: 0033:0x7f1a63983cd8 [ 5552.194905] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5552.202950] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5552.206223] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5552.209251] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5552.212323] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5552.215539] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5552.218329] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5556.062429] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5556.063083] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5556.066415] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5556.066495] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5556.066500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5556.066503] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5556.066517] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5556.066521] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5556.066526] RAX: 0000000000005c7e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5556.066528] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5556.066531] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5556.066533] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5556.066535] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5556.066539] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5556.066542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5556.066545] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5556.066554] Call Trace: [ 5556.066560] [ 5556.066565] ? show_regs.cold.9+0x22/0x2f [ 5556.066577] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5556.066584] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5556.066588] ? watchdog+0x40/0x40 [ 5556.066592] ? hrtimer_interrupt+0x128/0x2c0 [ 5556.066596] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5556.066604] ? apic_timer_interrupt+0xf/0x20 [ 5556.066609] [ 5556.066612] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5556.066619] do_raw_spin_lock+0x177/0x190 [ 5556.066624] _raw_spin_lock+0x21/0x30 [ 5556.066629] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5556.067373] ? do_raw_spin_unlock+0x75/0x190 [ 5556.067379] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5556.067644] ? do_wait_intr+0xf0/0xf0 [ 5556.067651] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5556.067872] mdc_close+0x53b/0xd70 [mdc] [ 5556.081386] Modules linked in: lustre(O) [ 5556.109606] lmv_close+0x18d/0x490 [lmv] [ 5556.109648] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5556.109847] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5556.109933] ll_md_close+0x274/0x800 [lustre] [ 5556.110259] ll_file_release+0x128/0xc50 [lustre] [ 5556.110345] __fput+0x10d/0x440 [ 5556.110355] ____fput+0x12/0x20 [ 5556.110360] task_work_run+0xc6/0x110 [ 5556.110371] exit_to_usermode_loop+0x1dd/0x1f0 [ 5556.110379] do_syscall_64+0x3d6/0x440 [ 5556.110384] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5556.110393] RIP: 0033:0x7f1a63983cd8 [ 5556.110399] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5556.110404] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5556.110411] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5556.119372] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5556.119449] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5556.119457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5556.119461] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5556.119475] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5556.119479] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5556.119484] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5556.119486] RDX: 00000000000010ff RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5556.119488] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5556.119490] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5556.119492] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5556.119496] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5556.119499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5556.119501] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5556.119510] Call Trace: [ 5556.119514] [ 5556.119520] ? show_regs.cold.9+0x22/0x2f [ 5556.119531] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5556.119538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5556.119543] ? watchdog+0x40/0x40 [ 5556.119547] ? hrtimer_interrupt+0x128/0x2c0 [ 5556.119551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5556.119559] ? apic_timer_interrupt+0xf/0x20 [ 5556.119564] [ 5556.119567] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5556.119572] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5556.119576] do_raw_spin_lock+0x177/0x190 [ 5556.119581] _raw_spin_lock+0x21/0x30 [ 5556.119586] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5556.119856] obd_mod_free+0x46/0x140 [ptlrpc] [ 5556.121141] mdc_commit_open+0x91/0xb0 [mdc] [ 5556.121178] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5556.121415] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5556.121638] after_reply+0xe2c/0x1780 [ptlrpc] [ 5556.121859] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5556.129950] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5556.129960] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5556.129962] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5556.129964] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5556.603597] ? __next_timer_interrupt+0x160/0x160 [ 5556.616138] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5556.617690] ? do_wait_intr+0xf0/0xf0 [ 5556.618900] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5556.627087] kthread+0x1d1/0x200 [ 5556.629997] ? set_kthread_struct+0x70/0x70 [ 5556.634232] ret_from_fork+0x1f/0x30 [ 5560.053571] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 5560.060154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5560.099730] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5560.115666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5560.126353] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5560.136850] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5560.161213] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5560.166583] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5560.170975] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5560.176407] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5560.180357] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5560.184777] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5560.189835] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5560.196542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5560.203364] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5560.207947] Call Trace: [ 5560.208825] [ 5560.209513] ? show_regs.cold.9+0x22/0x2f [ 5560.211761] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5560.213433] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5560.218142] ? watchdog+0x40/0x40 [ 5560.219710] ? hrtimer_interrupt+0x128/0x2c0 [ 5560.224747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5560.230337] ? apic_timer_interrupt+0xf/0x20 [ 5560.236596] [ 5560.237808] ? native_safe_halt+0x2e/0x50 [ 5560.242173] kvm_wait+0xcd/0xe0 [ 5560.245415] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5560.250175] do_raw_spin_lock+0x177/0x190 [ 5560.254056] _raw_spin_lock+0x21/0x30 [ 5560.256463] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5560.259658] ? do_raw_spin_unlock+0x75/0x190 [ 5560.262121] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5560.265592] ? do_wait_intr+0xf0/0xf0 [ 5560.268621] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5560.271572] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5560.274924] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5560.278860] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5560.282404] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5560.285503] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5560.288462] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5560.292706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5560.295585] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5560.299339] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5560.302072] lmv_intent_open+0x338/0x1150 [lmv] [ 5560.304275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5560.306770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5560.311862] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5560.314237] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5560.316811] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5560.319712] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5560.326866] ? do_raw_spin_unlock+0x75/0x190 [ 5560.328838] ? _raw_spin_unlock+0x12/0x30 [ 5560.330923] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5560.334630] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5560.339086] ll_atomic_open+0x36a/0xd70 [lustre] [ 5560.341707] lookup_open+0x356/0xb40 [ 5560.343624] do_last+0x313/0xfc0 [ 5560.345537] ? nd_jump_root+0xe5/0x160 [ 5560.347529] ? path_init+0x437/0x520 [ 5560.349142] path_openat+0xf7/0x500 [ 5560.350750] ? do_raw_spin_unlock+0x75/0x190 [ 5560.352686] do_filp_open+0x99/0x140 [ 5560.354417] ? getname_flags+0x6e/0x330 [ 5560.356457] ? __check_object_size+0xff/0x256 [ 5560.358927] ? do_raw_spin_unlock+0x75/0x190 [ 5560.362976] ? _raw_spin_unlock+0x12/0x30 [ 5560.366919] do_sys_openat2+0x2b4/0x410 [ 5560.371526] do_sys_open+0x73/0xa0 [ 5560.372750] __x64_sys_openat+0x24/0x30 [ 5560.377571] do_syscall_64+0xc1/0x440 [ 5560.380717] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5560.384676] RIP: 0033:0x7f1a63983332 [ 5560.385982] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5560.396403] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5560.398789] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5560.402638] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5560.408176] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5560.410773] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5560.417911] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5563.876122] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3540s! [ 5563.881656] Showing busy workqueues and worker pools: [ 5563.883850] workqueue events: flags=0x0 [ 5563.885622] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5563.890612] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5563.904468] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5563.912549] pending: sec_gc_main [ptlrpc] [ 5563.923451] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5563.929550] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5563.941167] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5563.963665] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5563.980788] workqueue events_long: flags=0x0 [ 5563.983812] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5563.994580] pending: lu_objects_flush [obdclass] [ 5564.004775] workqueue events_unbound: flags=0x2 [ 5564.011551] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5564.015525] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5564.021889] workqueue events_power_efficient: flags=0x80 [ 5564.024635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5564.031225] pending: neigh_periodic_work, do_cache_clean [ 5564.034741] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5564.038877] pending: neigh_periodic_work, check_lifetime [ 5564.045082] workqueue mm_percpu_wq: flags=0x8 [ 5564.047634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5564.053760] pending: vmstat_update [ 5564.056142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5564.059940] pending: vmstat_update [ 5564.065320] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5564.073678] pending: vmstat_update [ 5564.079836] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5564.082059] pending: vmstat_update [ 5564.085068] workqueue writeback: flags=0x4a [ 5564.089359] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5564.099107] pending: wb_workfn, wb_workfn [ 5564.100816] workqueue xprtiod: flags=0xa [ 5564.110296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5564.116532] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5564.120294] workqueue ipv6_addrconf: flags=0x40008 [ 5564.122541] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5564.125233] pending: addrconf_verify_work [ 5564.127505] workqueue knbd0-recv: flags=0x1a [ 5564.131112] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5564.134526] in-flight: 647:recv_work [ 5564.136810] workqueue ptlrpc_pinger: flags=0x2 [ 5564.145157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5564.149715] pending: ptlrpc_pinger_main [ptlrpc] [ 5564.155105] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5554s workers=2 idle: 652 [ 5580.056078] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5580.065489] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5580.109691] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5580.114243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5580.117658] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5580.120098] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5580.128243] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5580.135992] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5580.142334] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5580.145415] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5580.153146] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5580.160135] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5580.169290] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5580.180055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5580.181934] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5580.189089] Call Trace: [ 5580.190543] [ 5580.192104] ? show_regs.cold.9+0x22/0x2f [ 5580.193406] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5580.198453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5580.207108] ? watchdog+0x40/0x40 [ 5580.210369] ? hrtimer_interrupt+0x128/0x2c0 [ 5580.216420] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5580.222403] ? apic_timer_interrupt+0xf/0x20 [ 5580.225039] [ 5580.226090] ? native_safe_halt+0x2e/0x50 [ 5580.229407] kvm_wait+0xcd/0xe0 [ 5580.232862] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5580.243186] do_raw_spin_lock+0x177/0x190 [ 5580.245172] _raw_spin_lock+0x21/0x30 [ 5580.246469] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5580.250637] ? do_raw_spin_unlock+0x75/0x190 [ 5580.252531] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5580.254512] ? do_wait_intr+0xf0/0xf0 [ 5580.255734] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5580.260675] mdc_close+0x53b/0xd70 [mdc] [ 5580.263272] lmv_close+0x18d/0x490 [lmv] [ 5580.264739] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5580.267731] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5580.274518] ll_md_close+0x274/0x800 [lustre] [ 5580.278599] ll_file_release+0x128/0xc50 [lustre] [ 5580.281238] __fput+0x10d/0x440 [ 5580.282433] ____fput+0x12/0x20 [ 5580.284604] task_work_run+0xc6/0x110 [ 5580.286368] exit_to_usermode_loop+0x1dd/0x1f0 [ 5580.288084] do_syscall_64+0x3d6/0x440 [ 5580.289462] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5580.291324] RIP: 0033:0x7f1a63983cd8 [ 5580.293529] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5580.305549] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5580.308727] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5580.312404] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5580.316504] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5580.320372] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5580.327974] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5584.060752] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5584.063404] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5584.068246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5584.068327] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5584.068339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5584.068342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5584.068367] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5584.068371] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5584.068376] RAX: 0000000000000c49 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5584.068378] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5584.068380] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5584.068382] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5584.068385] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5584.068388] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5584.068395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5584.068398] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5584.068405] Call Trace: [ 5584.068411] [ 5584.068417] ? show_regs.cold.9+0x22/0x2f [ 5584.068434] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5584.068442] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5584.068446] ? watchdog+0x40/0x40 [ 5584.068451] ? hrtimer_interrupt+0x128/0x2c0 [ 5584.068455] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5584.068470] ? apic_timer_interrupt+0xf/0x20 [ 5584.068475] [ 5584.068478] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5584.068484] do_raw_spin_lock+0x177/0x190 [ 5584.068489] _raw_spin_lock+0x21/0x30 [ 5584.068500] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5584.068906] ? do_raw_spin_unlock+0x75/0x190 [ 5584.068913] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5584.069128] ? do_wait_intr+0xf0/0xf0 [ 5584.069135] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5584.069477] mdc_close+0x53b/0xd70 [mdc] [ 5584.069512] lmv_close+0x18d/0x490 [lmv] [ 5584.069542] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5584.069672] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5584.069746] ll_md_close+0x274/0x800 [lustre] [ 5584.069842] ll_file_release+0x128/0xc50 [lustre] [ 5584.070085] __fput+0x10d/0x440 [ 5584.070096] ____fput+0x12/0x20 [ 5584.070100] task_work_run+0xc6/0x110 [ 5584.070108] exit_to_usermode_loop+0x1dd/0x1f0 [ 5584.070115] do_syscall_64+0x3d6/0x440 [ 5584.070120] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5584.070128] RIP: 0033:0x7f1a63983cd8 [ 5584.070136] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5584.070141] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5584.070145] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5584.070148] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5584.070150] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5584.070152] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5584.070153] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5584.302252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5584.336987] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5584.344175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5584.356070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5584.367787] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5584.392761] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5584.397513] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5584.403274] RDX: 0000000000002169 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5584.405748] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5584.408594] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5584.414846] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5584.421185] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5584.424708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5584.430165] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5584.434612] Call Trace: [ 5584.436700] [ 5584.437888] ? show_regs.cold.9+0x22/0x2f [ 5584.439994] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5584.445017] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5584.448142] ? watchdog+0x40/0x40 [ 5584.454032] ? hrtimer_interrupt+0x128/0x2c0 [ 5584.457259] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5584.460211] ? apic_timer_interrupt+0xf/0x20 [ 5584.463015] [ 5584.464256] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5584.467341] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5584.470376] do_raw_spin_lock+0x177/0x190 [ 5584.473358] _raw_spin_lock+0x21/0x30 [ 5584.479048] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5584.489869] obd_mod_free+0x46/0x140 [ptlrpc] [ 5584.494480] mdc_commit_open+0x91/0xb0 [mdc] [ 5584.501435] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5584.504359] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5584.512776] after_reply+0xe2c/0x1780 [ptlrpc] [ 5584.515142] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5584.518107] ? __next_timer_interrupt+0x160/0x160 [ 5584.521899] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5584.523231] ? do_wait_intr+0xf0/0xf0 [ 5584.524753] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5584.530074] kthread+0x1d1/0x200 [ 5584.531360] ? set_kthread_struct+0x70/0x70 [ 5584.533523] ret_from_fork+0x1f/0x30 [ 5592.053038] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 5592.056232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5592.123428] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5592.136076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5592.143496] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5592.151504] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5592.169274] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5592.172164] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5592.175187] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5592.178034] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5592.181267] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5592.195299] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5592.200401] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5592.206503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5592.215157] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5592.220244] Call Trace: [ 5592.222326] [ 5592.223500] ? show_regs.cold.9+0x22/0x2f [ 5592.228568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5592.235095] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5592.236611] ? watchdog+0x40/0x40 [ 5592.245398] ? hrtimer_interrupt+0x128/0x2c0 [ 5592.249495] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5592.253178] ? apic_timer_interrupt+0xf/0x20 [ 5592.259244] [ 5592.263396] ? native_safe_halt+0x2e/0x50 [ 5592.274100] kvm_wait+0xcd/0xe0 [ 5592.282542] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5592.289493] do_raw_spin_lock+0x177/0x190 [ 5592.293799] _raw_spin_lock+0x21/0x30 [ 5592.296659] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5592.306137] ? do_raw_spin_unlock+0x75/0x190 [ 5592.310073] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5592.314485] ? do_wait_intr+0xf0/0xf0 [ 5592.318158] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5592.324619] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5592.329882] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5592.334796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5592.345386] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5592.355248] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5592.361319] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5592.364294] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5592.371441] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5592.383569] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5592.388040] lmv_intent_open+0x338/0x1150 [lmv] [ 5592.397075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5592.400086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5592.407495] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5592.415256] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5592.422518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5592.429441] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5592.435959] ? do_raw_spin_unlock+0x75/0x190 [ 5592.442473] ? _raw_spin_unlock+0x12/0x30 [ 5592.445206] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5592.450047] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5592.458563] ll_atomic_open+0x36a/0xd70 [lustre] [ 5592.468798] lookup_open+0x356/0xb40 [ 5592.474956] do_last+0x313/0xfc0 [ 5592.478402] ? nd_jump_root+0xe5/0x160 [ 5592.480021] ? path_init+0x437/0x520 [ 5592.482176] path_openat+0xf7/0x500 [ 5592.487645] ? do_raw_spin_unlock+0x75/0x190 [ 5592.491551] do_filp_open+0x99/0x140 [ 5592.495518] ? getname_flags+0x6e/0x330 [ 5592.501634] ? __check_object_size+0xff/0x256 [ 5592.503804] ? do_raw_spin_unlock+0x75/0x190 [ 5592.507401] ? _raw_spin_unlock+0x12/0x30 [ 5592.511325] do_sys_openat2+0x2b4/0x410 [ 5592.513021] do_sys_open+0x73/0xa0 [ 5592.516678] __x64_sys_openat+0x24/0x30 [ 5592.523940] do_syscall_64+0xc1/0x440 [ 5592.530267] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5592.537157] RIP: 0033:0x7f1a63983332 [ 5592.543732] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5592.571008] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5592.576278] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5592.582549] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5592.585390] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5592.591081] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5592.597434] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5594.594058] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3571s! [ 5594.599144] Showing busy workqueues and worker pools: [ 5594.601416] workqueue events: flags=0x0 [ 5594.603218] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5594.606391] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5594.609929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5594.612945] pending: sec_gc_main [ptlrpc] [ 5594.615278] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5594.618167] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5594.622945] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5594.625915] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5594.630418] workqueue events_long: flags=0x0 [ 5594.632825] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5594.636167] pending: lu_objects_flush [obdclass] [ 5594.638833] workqueue events_unbound: flags=0x2 [ 5594.641228] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5594.644091] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5594.648957] workqueue events_power_efficient: flags=0x80 [ 5594.651778] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5594.655139] pending: neigh_periodic_work, do_cache_clean [ 5594.657610] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5594.660742] pending: neigh_periodic_work, check_lifetime [ 5594.663436] workqueue mm_percpu_wq: flags=0x8 [ 5594.665630] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5594.668856] pending: vmstat_update [ 5594.670937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5594.673514] pending: vmstat_update [ 5594.675464] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5594.678439] pending: vmstat_update [ 5594.680191] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5594.683186] pending: vmstat_update [ 5594.685073] workqueue writeback: flags=0x4a [ 5594.686990] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5594.689943] pending: wb_workfn, wb_workfn [ 5594.692239] workqueue xprtiod: flags=0xa [ 5594.694011] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5594.701432] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5594.705323] workqueue ipv6_addrconf: flags=0x40008 [ 5594.707832] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5594.711436] pending: addrconf_verify_work [ 5594.713406] workqueue knbd0-recv: flags=0x1a [ 5594.715567] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5594.718529] in-flight: 647:recv_work [ 5594.720468] workqueue ptlrpc_pinger: flags=0x2 [ 5594.722896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5594.725485] pending: ptlrpc_pinger_main [ptlrpc] [ 5594.728238] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5584s workers=2 idle: 652 [ 5608.058236] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5608.073827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5608.178211] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5608.193921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5608.212791] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5608.224079] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5608.256017] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5608.265589] RAX: 0000000000004190 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5608.277762] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5608.286578] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5608.298403] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5608.309056] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5608.323738] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5608.334194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5608.340379] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5608.349903] Call Trace: [ 5608.351252] [ 5608.352011] ? show_regs.cold.9+0x22/0x2f [ 5608.353474] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5608.355425] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5608.357394] ? watchdog+0x40/0x40 [ 5608.358806] ? hrtimer_interrupt+0x128/0x2c0 [ 5608.367734] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5608.372133] ? apic_timer_interrupt+0xf/0x20 [ 5608.376135] [ 5608.384655] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5608.388119] do_raw_spin_lock+0x177/0x190 [ 5608.390056] _raw_spin_lock+0x21/0x30 [ 5608.391874] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5608.394673] ? do_raw_spin_unlock+0x75/0x190 [ 5608.396731] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5608.399054] ? do_wait_intr+0xf0/0xf0 [ 5608.400697] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5608.407678] mdc_close+0x53b/0xd70 [mdc] [ 5608.411035] lmv_close+0x18d/0x490 [lmv] [ 5608.412403] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5608.422200] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5608.432100] ll_md_close+0x274/0x800 [lustre] [ 5608.433942] ll_file_release+0x128/0xc50 [lustre] [ 5608.439010] __fput+0x10d/0x440 [ 5608.442228] ____fput+0x12/0x20 [ 5608.445088] task_work_run+0xc6/0x110 [ 5608.449848] exit_to_usermode_loop+0x1dd/0x1f0 [ 5608.456142] do_syscall_64+0x3d6/0x440 [ 5608.457518] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5608.469820] RIP: 0033:0x7f1a63983cd8 [ 5608.473396] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5608.511308] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5608.523023] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5608.527099] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5608.545149] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5608.554033] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5608.565192] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5612.057545] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5612.058016] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5612.077223] Modules linked in: lustre(O) [ 5612.080638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5612.080718] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5612.080726] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5612.080729] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5612.080745] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5612.080750] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5612.080755] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5612.080757] RDX: 0000000000000800 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5612.080760] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5612.080762] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5612.080765] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5612.080768] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5612.080771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5612.080773] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5612.080783] Call Trace: [ 5612.080788] [ 5612.080794] ? show_regs.cold.9+0x22/0x2f [ 5612.080805] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5612.080811] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5612.080816] ? watchdog+0x40/0x40 [ 5612.080821] ? hrtimer_interrupt+0x128/0x2c0 [ 5612.080825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5612.080834] ? apic_timer_interrupt+0xf/0x20 [ 5612.080838] [ 5612.080841] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5612.080845] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5612.080850] do_raw_spin_lock+0x177/0x190 [ 5612.080854] _raw_spin_lock+0x21/0x30 [ 5612.080860] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5612.081148] obd_mod_free+0x46/0x140 [ptlrpc] [ 5612.081346] mdc_commit_open+0x91/0xb0 [mdc] [ 5612.081385] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5612.081681] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5612.081891] after_reply+0xe2c/0x1780 [ptlrpc] [ 5612.082097] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5612.082303] ? __next_timer_interrupt+0x160/0x160 [ 5612.082312] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5612.082678] ? do_wait_intr+0xf0/0xf0 [ 5612.082688] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5612.082892] kthread+0x1d1/0x200 [ 5612.082898] ? set_kthread_struct+0x70/0x70 [ 5612.082902] ret_from_fork+0x1f/0x30 [ 5612.314089] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5612.336574] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5612.344268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5612.350048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5612.353191] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5612.365006] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5612.371206] RAX: 0000000000004b83 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5612.380253] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5612.387384] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5612.395402] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5612.400792] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5612.406133] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5612.411876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5612.416223] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5612.423009] Call Trace: [ 5612.426009] [ 5612.426872] ? show_regs.cold.9+0x22/0x2f [ 5612.429428] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5612.433200] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5612.436253] ? watchdog+0x40/0x40 [ 5612.439207] ? hrtimer_interrupt+0x128/0x2c0 [ 5612.447212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5612.452886] ? apic_timer_interrupt+0xf/0x20 [ 5612.455964] [ 5612.456822] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5612.462501] do_raw_spin_lock+0x177/0x190 [ 5612.467297] _raw_spin_lock+0x21/0x30 [ 5612.471220] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5612.476183] ? do_raw_spin_unlock+0x75/0x190 [ 5612.482690] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5612.488559] ? do_wait_intr+0xf0/0xf0 [ 5612.490807] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5612.501656] mdc_close+0x53b/0xd70 [mdc] [ 5612.503832] lmv_close+0x18d/0x490 [lmv] [ 5612.506166] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5612.513562] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5612.518385] ll_md_close+0x274/0x800 [lustre] [ 5612.520477] ll_file_release+0x128/0xc50 [lustre] [ 5612.527453] __fput+0x10d/0x440 [ 5612.532123] ____fput+0x12/0x20 [ 5612.537122] task_work_run+0xc6/0x110 [ 5612.541647] exit_to_usermode_loop+0x1dd/0x1f0 [ 5612.544576] do_syscall_64+0x3d6/0x440 [ 5612.546393] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5612.549216] RIP: 0033:0x7f1a63983cd8 [ 5612.552142] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5612.561970] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5612.565770] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5612.569198] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5612.573169] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5612.577205] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5612.580412] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5620.053264] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 5620.059390] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5620.103285] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5620.111099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5620.116856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5620.122813] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 5620.131810] RSP: 0018:ffffc38541c37500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5620.136891] RAX: 0000000000002cdf RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5620.142655] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5620.145933] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5620.149214] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5620.152801] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5620.155792] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5620.159644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5620.165257] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5620.168700] Call Trace: [ 5620.170078] [ 5620.171277] ? show_regs.cold.9+0x22/0x2f [ 5620.172918] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5620.177003] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5620.180366] ? watchdog+0x40/0x40 [ 5620.183149] ? hrtimer_interrupt+0x128/0x2c0 [ 5620.185395] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5620.187772] ? apic_timer_interrupt+0xf/0x20 [ 5620.190026] [ 5620.191081] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5620.193256] do_raw_spin_lock+0x177/0x190 [ 5620.195318] _raw_spin_lock+0x21/0x30 [ 5620.196838] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5620.199891] ? do_raw_spin_unlock+0x75/0x190 [ 5620.201589] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5620.204784] ? do_wait_intr+0xf0/0xf0 [ 5620.206439] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5620.208907] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5620.211303] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5620.214755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5620.218597] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5620.222588] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5620.224841] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5620.227103] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5620.229161] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5620.231872] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5620.235511] lmv_intent_open+0x338/0x1150 [lmv] [ 5620.238939] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5620.242727] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5620.248239] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5620.253972] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5620.256289] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5620.259166] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5620.262083] ? do_raw_spin_unlock+0x75/0x190 [ 5620.264396] ? _raw_spin_unlock+0x12/0x30 [ 5620.266266] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5620.269175] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5620.272298] ll_atomic_open+0x36a/0xd70 [lustre] [ 5620.274540] lookup_open+0x356/0xb40 [ 5620.276261] do_last+0x313/0xfc0 [ 5620.278207] ? nd_jump_root+0xe5/0x160 [ 5620.280116] ? path_init+0x437/0x520 [ 5620.282312] path_openat+0xf7/0x500 [ 5620.284779] ? do_raw_spin_unlock+0x75/0x190 [ 5620.287061] do_filp_open+0x99/0x140 [ 5620.288505] ? getname_flags+0x6e/0x330 [ 5620.290602] ? __check_object_size+0xff/0x256 [ 5620.292760] ? do_raw_spin_unlock+0x75/0x190 [ 5620.294603] ? _raw_spin_unlock+0x12/0x30 [ 5620.297043] do_sys_openat2+0x2b4/0x410 [ 5620.298755] do_sys_open+0x73/0xa0 [ 5620.300173] __x64_sys_openat+0x24/0x30 [ 5620.302213] do_syscall_64+0xc1/0x440 [ 5620.303611] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5620.306133] RIP: 0033:0x7f1a63983332 [ 5620.307482] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5620.316888] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5620.321039] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5620.324676] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5620.328294] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5620.332058] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5620.335358] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5624.802096] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3601s! [ 5624.813407] Showing busy workqueues and worker pools: [ 5624.815142] workqueue events: flags=0x0 [ 5624.817901] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5624.822933] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5624.828618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5624.832359] pending: sec_gc_main [ptlrpc] [ 5624.834389] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5624.837274] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5624.842386] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5624.847839] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5624.853422] workqueue events_long: flags=0x0 [ 5624.855575] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5624.859298] pending: lu_objects_flush [obdclass] [ 5624.861806] workqueue events_unbound: flags=0x2 [ 5624.863834] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5624.869970] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5624.878879] workqueue events_power_efficient: flags=0x80 [ 5624.883105] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5624.893665] pending: neigh_periodic_work, do_cache_clean [ 5624.899858] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5624.904126] pending: neigh_periodic_work, check_lifetime [ 5624.907757] workqueue mm_percpu_wq: flags=0x8 [ 5624.910314] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5624.914460] pending: vmstat_update [ 5624.916714] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5624.920615] pending: vmstat_update [ 5624.922684] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5624.926493] pending: vmstat_update [ 5624.928469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5624.932156] pending: vmstat_update [ 5624.934416] workqueue writeback: flags=0x4a [ 5624.938018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5624.945965] pending: wb_workfn, wb_workfn [ 5624.951461] workqueue xprtiod: flags=0xa [ 5624.954971] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5624.957288] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5624.961470] workqueue ipv6_addrconf: flags=0x40008 [ 5624.964389] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5624.967967] pending: addrconf_verify_work [ 5624.970318] workqueue knbd0-recv: flags=0x1a [ 5624.972415] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5624.975753] in-flight: 647:recv_work [ 5624.977753] workqueue ptlrpc_pinger: flags=0x2 [ 5624.980188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5624.983028] pending: ptlrpc_pinger_main [ptlrpc] [ 5624.989065] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5615s workers=2 idle: 652 [ 5636.056059] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5636.059085] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5636.078946] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5636.083966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5636.087695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5636.090088] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5636.097893] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5636.100418] RAX: 0000000000002d40 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5636.102710] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5636.105357] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5636.107940] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5636.110700] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5636.113339] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5636.116209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5636.118353] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5636.120945] Call Trace: [ 5636.122109] [ 5636.123011] ? show_regs.cold.9+0x22/0x2f [ 5636.124681] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5636.126558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5636.128473] ? watchdog+0x40/0x40 [ 5636.129844] ? hrtimer_interrupt+0x128/0x2c0 [ 5636.131592] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5636.133462] ? apic_timer_interrupt+0xf/0x20 [ 5636.135123] [ 5636.136018] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5636.138359] do_raw_spin_lock+0x177/0x190 [ 5636.140099] _raw_spin_lock+0x21/0x30 [ 5636.141703] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5636.144415] ? do_raw_spin_unlock+0x75/0x190 [ 5636.145949] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5636.148221] ? do_wait_intr+0xf0/0xf0 [ 5636.149826] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5636.152311] mdc_close+0x53b/0xd70 [mdc] [ 5636.153752] lmv_close+0x18d/0x490 [lmv] [ 5636.155655] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5636.158628] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5636.160931] ll_md_close+0x274/0x800 [lustre] [ 5636.162506] ll_file_release+0x128/0xc50 [lustre] [ 5636.164658] __fput+0x10d/0x440 [ 5636.165828] ____fput+0x12/0x20 [ 5636.166970] task_work_run+0xc6/0x110 [ 5636.168117] exit_to_usermode_loop+0x1dd/0x1f0 [ 5636.169703] do_syscall_64+0x3d6/0x440 [ 5636.171965] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5636.176241] RIP: 0033:0x7f1a63983cd8 [ 5636.180665] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5636.189912] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5636.193895] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5636.199060] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5636.202511] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5636.205207] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5636.207019] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5640.057028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5640.058183] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5640.060437] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5640.060514] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5640.060522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5640.060525] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5640.060539] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5640.060543] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5640.060548] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5640.060550] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5640.060553] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5640.060555] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5640.060557] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5640.060560] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5640.060562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5640.060564] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5640.060573] Call Trace: [ 5640.060579] [ 5640.060585] ? show_regs.cold.9+0x22/0x2f [ 5640.060596] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5640.060602] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5640.060608] ? watchdog+0x40/0x40 [ 5640.060612] ? hrtimer_interrupt+0x128/0x2c0 [ 5640.060616] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5640.060623] ? apic_timer_interrupt+0xf/0x20 [ 5640.060627] [ 5640.060630] ? native_safe_halt+0x2e/0x50 [ 5640.060634] kvm_wait+0xcd/0xe0 [ 5640.060641] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5640.060649] do_raw_spin_lock+0x177/0x190 [ 5640.060654] _raw_spin_lock+0x21/0x30 [ 5640.060658] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5640.060922] ? do_raw_spin_unlock+0x75/0x190 [ 5640.060927] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5640.061116] ? do_wait_intr+0xf0/0xf0 [ 5640.061125] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5640.081977] Modules linked in: lustre(O) [ 5640.106059] mdc_close+0x53b/0xd70 [mdc] [ 5640.106108] lmv_close+0x18d/0x490 [lmv] [ 5640.106327] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5640.106436] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5640.106513] ll_md_close+0x274/0x800 [lustre] [ 5640.106587] ll_file_release+0x128/0xc50 [lustre] [ 5640.106662] __fput+0x10d/0x440 [ 5640.106672] ____fput+0x12/0x20 [ 5640.106676] task_work_run+0xc6/0x110 [ 5640.106683] exit_to_usermode_loop+0x1dd/0x1f0 [ 5640.106690] do_syscall_64+0x3d6/0x440 [ 5640.106694] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5640.106702] RIP: 0033:0x7f1a63983cd8 [ 5640.106708] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5640.106712] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5640.106717] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5640.106719] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5640.106721] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5640.106723] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5640.106725] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5640.397135] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5640.435411] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5640.440888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5640.450751] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5640.454211] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5640.472335] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5640.476819] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5640.480196] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5640.489517] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5640.496493] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5640.501808] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5640.513577] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5640.518650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5640.521007] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5640.524930] Call Trace: [ 5640.527778] [ 5640.530615] ? show_regs.cold.9+0x22/0x2f [ 5640.532686] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5640.535282] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5640.537358] ? watchdog+0x40/0x40 [ 5640.538482] ? hrtimer_interrupt+0x128/0x2c0 [ 5640.539844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5640.543554] ? apic_timer_interrupt+0xf/0x20 [ 5640.545010] [ 5640.546296] ? native_safe_halt+0x2e/0x50 [ 5640.547943] kvm_wait+0xcd/0xe0 [ 5640.549768] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5640.551777] do_raw_spin_lock+0x177/0x190 [ 5640.562509] _raw_spin_lock+0x21/0x30 [ 5640.565712] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5640.567614] obd_mod_free+0x46/0x140 [ptlrpc] [ 5640.573015] mdc_commit_open+0x91/0xb0 [mdc] [ 5640.576783] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5640.582012] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5640.590264] after_reply+0xe2c/0x1780 [ptlrpc] [ 5640.592045] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5640.607423] ? __next_timer_interrupt+0x160/0x160 [ 5640.614231] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5640.615649] ? do_wait_intr+0xf0/0xf0 [ 5640.619405] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5640.624837] kthread+0x1d1/0x200 [ 5640.626594] ? set_kthread_struct+0x70/0x70 [ 5640.629110] ret_from_fork+0x1f/0x30 [ 5652.053791] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5652.060863] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5652.102641] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5652.111474] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5652.117683] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5652.127998] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5652.143545] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5652.146923] RAX: 0000000000007d8f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5652.149926] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5652.154282] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5652.160767] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5652.167773] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5652.176429] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5652.181422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5652.184892] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5652.188897] Call Trace: [ 5652.190791] [ 5652.191969] ? show_regs.cold.9+0x22/0x2f [ 5652.194426] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5652.197266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5652.199928] ? watchdog+0x40/0x40 [ 5652.201983] ? hrtimer_interrupt+0x128/0x2c0 [ 5652.204468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5652.207653] ? apic_timer_interrupt+0xf/0x20 [ 5652.210412] [ 5652.211748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5652.223558] do_raw_spin_lock+0x177/0x190 [ 5652.228659] _raw_spin_lock+0x21/0x30 [ 5652.229757] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5652.235792] ? do_raw_spin_unlock+0x75/0x190 [ 5652.238861] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5652.242239] ? do_wait_intr+0xf0/0xf0 [ 5652.243761] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5652.246250] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5652.249127] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5652.252028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5652.254283] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5652.262547] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5652.268705] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5652.271453] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5652.273668] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5652.277626] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5652.281076] lmv_intent_open+0x338/0x1150 [lmv] [ 5652.284247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5652.290001] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5652.300247] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5652.306370] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5652.312205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5652.315673] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5652.319982] ? do_raw_spin_unlock+0x75/0x190 [ 5652.325645] ? _raw_spin_unlock+0x12/0x30 [ 5652.331383] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5652.334415] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5652.341867] ll_atomic_open+0x36a/0xd70 [lustre] [ 5652.345226] lookup_open+0x356/0xb40 [ 5652.349214] do_last+0x313/0xfc0 [ 5652.351609] ? nd_jump_root+0xe5/0x160 [ 5652.354211] ? path_init+0x437/0x520 [ 5652.357009] path_openat+0xf7/0x500 [ 5652.359689] ? do_raw_spin_unlock+0x75/0x190 [ 5652.369680] do_filp_open+0x99/0x140 [ 5652.377083] ? getname_flags+0x6e/0x330 [ 5652.383932] ? __check_object_size+0xff/0x256 [ 5652.387460] ? do_raw_spin_unlock+0x75/0x190 [ 5652.391414] ? _raw_spin_unlock+0x12/0x30 [ 5652.395243] do_sys_openat2+0x2b4/0x410 [ 5652.397469] do_sys_open+0x73/0xa0 [ 5652.404212] __x64_sys_openat+0x24/0x30 [ 5652.407171] do_syscall_64+0xc1/0x440 [ 5652.410089] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5652.413255] RIP: 0033:0x7f1a63983332 [ 5652.415691] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5652.436511] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5652.446182] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5652.459339] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5652.465258] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5652.476685] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5652.484728] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5655.015791] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3632s! [ 5655.029118] Showing busy workqueues and worker pools: [ 5655.031023] workqueue events: flags=0x0 [ 5655.039100] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5655.046837] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5655.054399] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5655.061070] pending: sec_gc_main [ptlrpc] [ 5655.063942] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5655.079664] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5655.091042] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5655.103023] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5655.125369] workqueue events_long: flags=0x0 [ 5655.127039] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5655.139426] pending: lu_objects_flush [obdclass] [ 5655.146789] workqueue events_unbound: flags=0x2 [ 5655.149823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5655.157094] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5655.166312] workqueue events_power_efficient: flags=0x80 [ 5655.169914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5655.173880] pending: neigh_periodic_work, do_cache_clean [ 5655.178020] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5655.184645] pending: neigh_periodic_work, check_lifetime [ 5655.187542] workqueue mm_percpu_wq: flags=0x8 [ 5655.190728] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5655.199629] pending: vmstat_update [ 5655.201455] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5655.205417] pending: vmstat_update [ 5655.208419] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5655.218628] pending: vmstat_update [ 5655.226010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5655.237601] pending: vmstat_update [ 5655.242736] workqueue writeback: flags=0x4a [ 5655.246395] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5655.254324] pending: wb_workfn, wb_workfn [ 5655.260507] workqueue xprtiod: flags=0xa [ 5655.261956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5655.274058] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5655.280823] workqueue ipv6_addrconf: flags=0x40008 [ 5655.289064] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5655.298299] pending: addrconf_verify_work [ 5655.304145] workqueue knbd0-recv: flags=0x1a [ 5655.308424] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5655.312398] in-flight: 647:recv_work [ 5655.314792] workqueue ptlrpc_pinger: flags=0x2 [ 5655.316937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5655.321137] pending: ptlrpc_pinger_main [ptlrpc] [ 5655.324237] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5645s workers=2 idle: 652 [ 5664.056125] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 5664.060558] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5664.114898] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5664.134599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5664.144734] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5664.150829] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5664.176529] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5664.186218] RAX: 0000000000002bda RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5664.192781] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5664.197351] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5664.201115] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5664.216291] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5664.235302] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5664.247329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5664.251728] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5664.261133] Call Trace: [ 5664.261980] [ 5664.262711] ? show_regs.cold.9+0x22/0x2f [ 5664.267052] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5664.268810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5664.272663] ? watchdog+0x40/0x40 [ 5664.278892] ? hrtimer_interrupt+0x128/0x2c0 [ 5664.282912] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5664.288514] ? apic_timer_interrupt+0xf/0x20 [ 5664.291283] [ 5664.292682] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5664.295667] do_raw_spin_lock+0x177/0x190 [ 5664.298255] _raw_spin_lock+0x21/0x30 [ 5664.306896] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5664.318572] ? do_raw_spin_unlock+0x75/0x190 [ 5664.324597] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5664.332175] ? do_wait_intr+0xf0/0xf0 [ 5664.336125] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5664.342770] mdc_close+0x53b/0xd70 [mdc] [ 5664.358021] lmv_close+0x18d/0x490 [lmv] [ 5664.365831] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5664.375694] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5664.388681] ll_md_close+0x274/0x800 [lustre] [ 5664.401445] ll_file_release+0x128/0xc50 [lustre] [ 5664.424754] __fput+0x10d/0x440 [ 5664.428229] ____fput+0x12/0x20 [ 5664.429430] task_work_run+0xc6/0x110 [ 5664.446083] exit_to_usermode_loop+0x1dd/0x1f0 [ 5664.450624] do_syscall_64+0x3d6/0x440 [ 5664.465734] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5664.474308] RIP: 0033:0x7f1a63983cd8 [ 5664.476243] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5664.527327] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5664.547372] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5664.556862] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5664.568957] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5664.579315] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5664.596468] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5668.057041] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5668.058071] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5668.069808] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [ 5668.079016] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5668.079109] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5668.079115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5668.079118] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5668.079132] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5668.079137] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5668.079141] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5668.079143] RDX: 0000000000006f64 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5668.079145] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5668.079147] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5668.079150] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5668.079153] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5668.079156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5668.079159] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5668.079168] Call Trace: [ 5668.079483] [ 5668.079491] ? show_regs.cold.9+0x22/0x2f [ 5668.079513] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5668.079525] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5668.079530] ? watchdog+0x40/0x40 [ 5668.079535] ? hrtimer_interrupt+0x128/0x2c0 [ 5668.079539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5668.079549] ? apic_timer_interrupt+0xf/0x20 [ 5668.079554] [ 5668.079557] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5668.079563] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5668.079568] do_raw_spin_lock+0x177/0x190 [ 5668.079573] _raw_spin_lock+0x21/0x30 [ 5668.079579] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5668.080791] obd_mod_free+0x46/0x140 [ptlrpc] [ 5668.081265] mdc_commit_open+0x91/0xb0 [mdc] [ 5668.081444] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5668.081967] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5668.082620] after_reply+0xe2c/0x1780 [ptlrpc] [ 5668.083003] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5668.084141] ? __next_timer_interrupt+0x160/0x160 [ 5668.084153] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5668.085501] ? do_wait_intr+0xf0/0xf0 [ 5668.085609] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5668.086296] kthread+0x1d1/0x200 [ 5668.086303] ? set_kthread_struct+0x70/0x70 [ 5668.086320] ret_from_fork+0x1f/0x30 [ 5668.426192] sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5668.444464] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5668.463899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5668.476485] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5668.487158] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5668.513261] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5668.519572] RAX: 0000000000007c92 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5668.528698] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5668.533906] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5668.543428] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5668.550802] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5668.557489] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5668.565992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5668.574069] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5668.581436] Call Trace: [ 5668.583152] [ 5668.584679] ? show_regs.cold.9+0x22/0x2f [ 5668.587497] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5668.590929] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5668.593925] ? watchdog+0x40/0x40 [ 5668.596473] ? hrtimer_interrupt+0x128/0x2c0 [ 5668.600153] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5668.603427] ? apic_timer_interrupt+0xf/0x20 [ 5668.606407] [ 5668.608119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5668.613122] do_raw_spin_lock+0x177/0x190 [ 5668.615782] _raw_spin_lock+0x21/0x30 [ 5668.618397] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5668.622434] ? do_raw_spin_unlock+0x75/0x190 [ 5668.625556] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5668.629678] ? do_wait_intr+0xf0/0xf0 [ 5668.632343] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5668.636309] mdc_close+0x53b/0xd70 [mdc] [ 5668.639618] lmv_close+0x18d/0x490 [lmv] [ 5668.642830] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5668.649614] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5668.653583] ll_md_close+0x274/0x800 [lustre] [ 5668.657371] ll_file_release+0x128/0xc50 [lustre] [ 5668.664487] __fput+0x10d/0x440 [ 5668.666508] ____fput+0x12/0x20 [ 5668.671834] task_work_run+0xc6/0x110 [ 5668.674143] exit_to_usermode_loop+0x1dd/0x1f0 [ 5668.681501] do_syscall_64+0x3d6/0x440 [ 5668.685206] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5668.690123] RIP: 0033:0x7f1a63983cd8 [ 5668.692541] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5668.707788] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5668.718693] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5668.730124] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5668.737368] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5668.741798] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5668.749435] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5680.055642] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5680.070451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5680.104723] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5680.116360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5680.122559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5680.126929] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5680.139245] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5680.143022] RAX: 0000000000000f81 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5680.148722] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5680.168619] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5680.175669] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5680.180303] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5680.183703] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5680.187666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5680.195152] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5680.202638] Call Trace: [ 5680.203651] [ 5680.206979] ? show_regs.cold.9+0x22/0x2f [ 5680.211034] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5680.221537] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5680.224779] ? watchdog+0x40/0x40 [ 5680.227203] ? hrtimer_interrupt+0x128/0x2c0 [ 5680.231182] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5680.235141] ? apic_timer_interrupt+0xf/0x20 [ 5680.239517] [ 5680.240514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5680.252074] do_raw_spin_lock+0x177/0x190 [ 5680.253469] _raw_spin_lock+0x21/0x30 [ 5680.254661] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5680.264609] ? do_raw_spin_unlock+0x75/0x190 [ 5680.267674] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5680.274801] ? do_wait_intr+0xf0/0xf0 [ 5680.279252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5680.284734] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5680.286710] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5680.295214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5680.299408] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5680.302708] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5680.305714] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5680.308072] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5680.310674] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5680.314612] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5680.319533] lmv_intent_open+0x338/0x1150 [lmv] [ 5680.321664] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5680.324150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5680.326335] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5680.328270] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5680.330254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5680.332685] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5680.334561] ? do_raw_spin_unlock+0x75/0x190 [ 5680.336449] ? _raw_spin_unlock+0x12/0x30 [ 5680.341546] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5680.344724] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5680.349527] ll_atomic_open+0x36a/0xd70 [lustre] [ 5680.354943] lookup_open+0x356/0xb40 [ 5680.358616] do_last+0x313/0xfc0 [ 5680.361401] ? nd_jump_root+0xe5/0x160 [ 5680.363236] ? path_init+0x437/0x520 [ 5680.368535] path_openat+0xf7/0x500 [ 5680.370363] ? do_raw_spin_unlock+0x75/0x190 [ 5680.373452] do_filp_open+0x99/0x140 [ 5680.376622] ? getname_flags+0x6e/0x330 [ 5680.378433] ? __check_object_size+0xff/0x256 [ 5680.385552] ? do_raw_spin_unlock+0x75/0x190 [ 5680.389375] ? _raw_spin_unlock+0x12/0x30 [ 5680.391985] do_sys_openat2+0x2b4/0x410 [ 5680.394385] do_sys_open+0x73/0xa0 [ 5680.395978] __x64_sys_openat+0x24/0x30 [ 5680.400526] do_syscall_64+0xc1/0x440 [ 5680.402340] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5680.406645] RIP: 0033:0x7f1a63983332 [ 5680.414125] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5680.433980] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5680.444999] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5680.454654] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5680.464508] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5680.468604] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5680.473621] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5682.490020] rcu: INFO: rcu_sched self-detected stall on CPU [ 5682.492208] rcu: 3-...!: (2232789 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 5682.496982] (t=3660089 jiffies g=269633 q=1422) [ 5682.499157] rcu: rcu_sched kthread starved for 3659739 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 5682.503613] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5682.507600] rcu: RCU grace-period kthread stack dump: [ 5682.511627] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5682.516989] Call Trace: [ 5682.518476] __schedule+0x351/0xcb0 [ 5682.519631] schedule+0xc0/0x180 [ 5682.521196] schedule_timeout+0xb4/0x190 [ 5682.523032] ? do_raw_spin_unlock+0x75/0x190 [ 5682.524871] ? __next_timer_interrupt+0x160/0x160 [ 5682.527039] ? prepare_to_swait_event+0xf3/0x140 [ 5682.528617] rcu_gp_kthread+0x8dd/0x15b0 [ 5682.530449] ? force_qs_rnp+0x2c0/0x2c0 [ 5682.532050] kthread+0x1d1/0x200 [ 5682.533030] ? set_kthread_struct+0x70/0x70 [ 5682.534499] ret_from_fork+0x1f/0x30 [ 5682.536351] rcu: Stack dump where RCU GP kthread last ran: [ 5682.538323] Sending NMI from CPU 3 to CPUs 1: [ 5682.541024] NMI backtrace for cpu 1 [ 5682.541030] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5682.541031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5682.541033] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5682.541034] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 5682.541037] RSP: 0018:ffffc38542073af0 EFLAGS: 00000216 [ 5682.541040] RAX: 0000000000003709 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5682.541041] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5682.541043] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5682.541044] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5682.541046] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5682.541047] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5682.541048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5682.541049] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5682.541050] Call Trace: [ 5682.541051] [ 5682.541052] ? show_regs.cold.9+0x22/0x2f [ 5682.541053] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5682.541054] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5682.541055] ? nmi_handle+0xbd/0x210 [ 5682.541056] ? default_do_nmi+0x4b/0x1b0 [ 5682.541057] ? do_nmi+0x280/0x380 [ 5682.541058] ? end_repeat_nmi+0x16/0x1f [ 5682.541059] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5682.541061] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5682.541062] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5682.541063] [ 5682.541064] do_raw_spin_lock+0x177/0x190 [ 5682.541065] _raw_spin_lock+0x21/0x30 [ 5682.541066] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5682.541067] ? do_raw_spin_unlock+0x75/0x190 [ 5682.541068] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5682.541069] ? do_wait_intr+0xf0/0xf0 [ 5682.541070] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5682.541071] mdc_close+0x53b/0xd70 [mdc] [ 5682.541072] lmv_close+0x18d/0x490 [lmv] [ 5682.541073] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5682.541074] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5682.541075] ll_md_close+0x274/0x800 [lustre] [ 5682.541076] ll_file_release+0x128/0xc50 [lustre] [ 5682.541077] __fput+0x10d/0x440 [ 5682.541078] ____fput+0x12/0x20 [ 5682.541079] task_work_run+0xc6/0x110 [ 5682.541080] exit_to_usermode_loop+0x1dd/0x1f0 [ 5682.541081] do_syscall_64+0x3d6/0x440 [ 5682.541082] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5682.541083] RIP: 0033:0x7f1a63983cd8 [ 5682.541085] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5682.541087] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5682.541089] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5682.541091] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5682.541093] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5682.541094] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5682.541095] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5682.541255] NMI backtrace for cpu 3 [ 5682.667194] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5682.671441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5682.675740] Call Trace: [ 5682.677408] [ 5682.678216] ? dump_stack+0xbb/0x10e [ 5682.679678] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5682.695418] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5682.697385] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5682.703182] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5682.710241] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5682.713052] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5682.715690] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5682.718093] ? update_process_times+0x6d/0xc0 [ 5682.719701] ? tick_sched_handle.isra.18+0x27/0x90 [ 5682.721259] ? tick_sched_timer+0x53/0xb0 [ 5682.723024] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5682.724565] ? tick_sched_handle.isra.18+0x90/0x90 [ 5682.726732] ? hrtimer_interrupt+0x128/0x2c0 [ 5682.728388] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5682.730101] ? apic_timer_interrupt+0xf/0x20 [ 5682.731674] [ 5682.732613] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5682.734517] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5682.736481] ? do_raw_spin_lock+0x177/0x190 [ 5682.738143] ? _raw_spin_lock+0x21/0x30 [ 5682.739986] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5682.742548] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5682.744440] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5682.746436] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5682.748749] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5682.751484] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5682.753639] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5682.758551] ? __next_timer_interrupt+0x160/0x160 [ 5682.761726] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5682.767211] ? do_wait_intr+0xf0/0xf0 [ 5682.768987] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5682.772750] ? kthread+0x1d1/0x200 [ 5682.778075] ? set_kthread_struct+0x70/0x70 [ 5682.779557] ? ret_from_fork+0x1f/0x30 [ 5685.731206] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3662s! [ 5685.737415] Showing busy workqueues and worker pools: [ 5685.742204] workqueue events: flags=0x0 [ 5685.743592] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5685.755328] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5685.764373] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5685.769016] pending: sec_gc_main [ptlrpc] [ 5685.775401] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5685.779957] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5685.785977] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5685.793046] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5685.801121] workqueue events_long: flags=0x0 [ 5685.805236] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5685.817045] pending: lu_objects_flush [obdclass] [ 5685.825954] workqueue events_unbound: flags=0x2 [ 5685.831163] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5685.834677] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5685.838955] workqueue events_power_efficient: flags=0x80 [ 5685.840625] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5685.843753] pending: neigh_periodic_work, do_cache_clean [ 5685.847204] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5685.855698] pending: neigh_periodic_work, check_lifetime [ 5685.857825] workqueue mm_percpu_wq: flags=0x8 [ 5685.862318] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5685.864390] pending: vmstat_update [ 5685.869617] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5685.878519] pending: vmstat_update [ 5685.881716] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5685.886424] pending: vmstat_update [ 5685.889442] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5685.893641] pending: vmstat_update [ 5685.896055] workqueue writeback: flags=0x4a [ 5685.900678] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5685.903559] pending: wb_workfn, wb_workfn [ 5685.907199] workqueue xprtiod: flags=0xa [ 5685.909062] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5685.913489] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5685.922408] workqueue ipv6_addrconf: flags=0x40008 [ 5685.925158] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5685.928222] pending: addrconf_verify_work [ 5685.930689] workqueue knbd0-recv: flags=0x1a [ 5685.937491] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5685.941667] in-flight: 647:recv_work [ 5685.943731] workqueue ptlrpc_pinger: flags=0x2 [ 5685.946191] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5685.949737] pending: ptlrpc_pinger_main [ptlrpc] [ 5685.952627] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5675s workers=2 idle: 652 [ 5696.059374] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5696.063164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5696.093260] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5696.105832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5696.112818] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5696.115871] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5696.128476] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5696.131843] RAX: 0000000000001396 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5696.134647] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5696.137425] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5696.140340] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5696.142921] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5696.145619] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5696.149080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5696.151949] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5696.155107] Call Trace: [ 5696.156107] [ 5696.156950] ? show_regs.cold.9+0x22/0x2f [ 5696.165174] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5696.166967] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5696.171779] ? watchdog+0x40/0x40 [ 5696.172723] ? hrtimer_interrupt+0x128/0x2c0 [ 5696.174799] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5696.176866] ? apic_timer_interrupt+0xf/0x20 [ 5696.180002] [ 5696.181317] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5696.186484] do_raw_spin_lock+0x177/0x190 [ 5696.189075] _raw_spin_lock+0x21/0x30 [ 5696.193223] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5696.201631] ? do_raw_spin_unlock+0x75/0x190 [ 5696.203813] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5696.210821] ? do_wait_intr+0xf0/0xf0 [ 5696.212413] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5696.217823] mdc_close+0x53b/0xd70 [mdc] [ 5696.219650] lmv_close+0x18d/0x490 [lmv] [ 5696.223482] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5696.231907] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5696.242394] ll_md_close+0x274/0x800 [lustre] [ 5696.251179] ll_file_release+0x128/0xc50 [lustre] [ 5696.254039] __fput+0x10d/0x440 [ 5696.258415] ____fput+0x12/0x20 [ 5696.265633] task_work_run+0xc6/0x110 [ 5696.268090] exit_to_usermode_loop+0x1dd/0x1f0 [ 5696.275384] do_syscall_64+0x3d6/0x440 [ 5696.279062] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5696.285204] RIP: 0033:0x7f1a63983cd8 [ 5696.290657] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5696.309485] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5696.316488] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5696.319465] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5696.329042] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5696.337282] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5696.346232] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5708.056343] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 5708.058013] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 5708.065494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [ 5708.069095] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5708.069279] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5708.069286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5708.069289] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5708.069307] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5708.069311] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5708.069316] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5708.069318] RDX: 0000000000001a8c RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5708.069321] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5708.069323] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5708.069325] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5708.069328] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5708.069331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5708.069333] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5708.069341] Call Trace: [ 5708.069345] [ 5708.069351] ? show_regs.cold.9+0x22/0x2f [ 5708.069361] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5708.069366] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5708.069370] ? watchdog+0x40/0x40 [ 5708.069380] ? hrtimer_interrupt+0x128/0x2c0 [ 5708.069385] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5708.069393] ? apic_timer_interrupt+0xf/0x20 [ 5708.069397] [ 5708.069399] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5708.069404] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5708.069408] do_raw_spin_lock+0x177/0x190 [ 5708.069413] _raw_spin_lock+0x21/0x30 [ 5708.069418] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5708.069911] obd_mod_free+0x46/0x140 [ptlrpc] [ 5708.070119] mdc_commit_open+0x91/0xb0 [mdc] [ 5708.070153] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5708.070435] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5708.070880] after_reply+0xe2c/0x1780 [ptlrpc] [ 5708.071085] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5708.071411] ? __next_timer_interrupt+0x160/0x160 [ 5708.071427] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5708.083384] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [ 5708.109986] ? do_wait_intr+0xf0/0xf0 [ 5708.110010] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5708.119367] dm_mod sha512_ssse3 sha512_generic [ 5708.119386] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5708.119393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5708.119396] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5708.125284] kthread+0x1d1/0x200 [ 5708.125304] ? set_kthread_struct+0x70/0x70 [ 5708.125309] ret_from_fork+0x1f/0x30 [ 5708.307791] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5708.320762] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5708.323881] RAX: 0000000000007719 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5708.328149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5708.337754] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5708.344266] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5708.349230] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5708.358235] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5708.363122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5708.366119] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5708.370037] Call Trace: [ 5708.376542] [ 5708.377697] ? show_regs.cold.9+0x22/0x2f [ 5708.380163] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5708.382276] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5708.384534] ? watchdog+0x40/0x40 [ 5708.387393] ? hrtimer_interrupt+0x128/0x2c0 [ 5708.392371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5708.412015] ? apic_timer_interrupt+0xf/0x20 [ 5708.413574] [ 5708.414143] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5708.424658] do_raw_spin_lock+0x177/0x190 [ 5708.429475] _raw_spin_lock+0x21/0x30 [ 5708.431204] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5708.440854] ? do_raw_spin_unlock+0x75/0x190 [ 5708.445829] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5708.448027] ? do_wait_intr+0xf0/0xf0 [ 5708.452166] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5708.459845] mdc_close+0x53b/0xd70 [mdc] [ 5708.461543] lmv_close+0x18d/0x490 [lmv] [ 5708.467366] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5708.471441] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5708.479481] ll_md_close+0x274/0x800 [lustre] [ 5708.485446] ll_file_release+0x128/0xc50 [lustre] [ 5708.493913] __fput+0x10d/0x440 [ 5708.499449] ____fput+0x12/0x20 [ 5708.502413] task_work_run+0xc6/0x110 [ 5708.505444] exit_to_usermode_loop+0x1dd/0x1f0 [ 5708.511225] do_syscall_64+0x3d6/0x440 [ 5708.515488] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5708.518272] RIP: 0033:0x7f1a63983cd8 [ 5708.519812] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5708.543954] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5708.548048] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5708.556793] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5708.561587] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5708.567384] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5708.569157] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5712.053095] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5712.056545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5712.087476] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5712.102235] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5712.108452] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5712.111200] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5712.131280] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5712.133820] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5712.138246] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5712.144872] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5712.149380] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5712.164255] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5712.174454] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5712.186850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5712.197222] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5712.202874] Call Trace: [ 5712.207335] [ 5712.209909] ? show_regs.cold.9+0x22/0x2f [ 5712.215296] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5712.220618] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5712.223662] ? watchdog+0x40/0x40 [ 5712.227154] ? hrtimer_interrupt+0x128/0x2c0 [ 5712.235731] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5712.245268] ? apic_timer_interrupt+0xf/0x20 [ 5712.247946] [ 5712.250651] ? native_safe_halt+0x2e/0x50 [ 5712.261116] kvm_wait+0xcd/0xe0 [ 5712.267914] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5712.270733] do_raw_spin_lock+0x177/0x190 [ 5712.274734] _raw_spin_lock+0x21/0x30 [ 5712.277734] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5712.280892] ? do_raw_spin_unlock+0x75/0x190 [ 5712.282771] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5712.284542] ? do_wait_intr+0xf0/0xf0 [ 5712.285781] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5712.287307] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5712.288790] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5712.290350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.291811] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5712.293414] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5712.295330] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5712.297594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.299742] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5712.303689] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5712.308838] lmv_intent_open+0x338/0x1150 [lmv] [ 5712.313451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.315899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.318202] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5712.320200] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5712.322725] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.324233] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5712.326145] ? do_raw_spin_unlock+0x75/0x190 [ 5712.328153] ? _raw_spin_unlock+0x12/0x30 [ 5712.330057] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5712.340448] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5712.342313] ll_atomic_open+0x36a/0xd70 [lustre] [ 5712.358722] lookup_open+0x356/0xb40 [ 5712.365079] do_last+0x313/0xfc0 [ 5712.370385] ? nd_jump_root+0xe5/0x160 [ 5712.374385] ? path_init+0x437/0x520 [ 5712.377098] path_openat+0xf7/0x500 [ 5712.378520] ? do_raw_spin_unlock+0x75/0x190 [ 5712.383043] do_filp_open+0x99/0x140 [ 5712.386873] ? getname_flags+0x6e/0x330 [ 5712.388507] ? __check_object_size+0xff/0x256 [ 5712.391170] ? do_raw_spin_unlock+0x75/0x190 [ 5712.397972] ? _raw_spin_unlock+0x12/0x30 [ 5712.402675] do_sys_openat2+0x2b4/0x410 [ 5712.407386] do_sys_open+0x73/0xa0 [ 5712.410075] __x64_sys_openat+0x24/0x30 [ 5712.412154] do_syscall_64+0xc1/0x440 [ 5712.413538] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5712.420330] RIP: 0033:0x7f1a63983332 [ 5712.421786] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5712.436713] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5712.439971] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5712.443329] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5712.446366] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5712.450779] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5712.457141] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5716.450595] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3693s! [ 5716.456993] Showing busy workqueues and worker pools: [ 5716.459730] workqueue events: flags=0x0 [ 5716.463866] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5716.468315] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5716.474759] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5716.478304] pending: sec_gc_main [ptlrpc] [ 5716.483407] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5716.491433] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5716.506138] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5716.510735] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5716.516471] workqueue events_long: flags=0x0 [ 5716.518286] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5716.521061] pending: lu_objects_flush [obdclass] [ 5716.523919] workqueue events_unbound: flags=0x2 [ 5716.525697] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5716.529456] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5716.534437] workqueue events_power_efficient: flags=0x80 [ 5716.537106] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5716.540709] pending: neigh_periodic_work, do_cache_clean [ 5716.543743] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5716.546254] pending: neigh_periodic_work, check_lifetime [ 5716.554440] workqueue mm_percpu_wq: flags=0x8 [ 5716.559174] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5716.572696] pending: vmstat_update [ 5716.580072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5716.590411] pending: vmstat_update [ 5716.599364] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5716.608310] pending: vmstat_update [ 5716.611470] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5716.619849] pending: vmstat_update [ 5716.625117] workqueue writeback: flags=0x4a [ 5716.628021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5716.635823] pending: wb_workfn, wb_workfn [ 5716.642907] workqueue xprtiod: flags=0xa [ 5716.645327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5716.654155] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5716.663877] workqueue ipv6_addrconf: flags=0x40008 [ 5716.670399] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5716.676944] pending: addrconf_verify_work [ 5716.682740] workqueue knbd0-recv: flags=0x1a [ 5716.687227] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5716.697632] in-flight: 647:recv_work [ 5716.699074] workqueue ptlrpc_pinger: flags=0x2 [ 5716.700358] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5716.710515] pending: ptlrpc_pinger_main [ptlrpc] [ 5716.721077] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5706s workers=2 idle: 652 [ 5724.061873] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5724.080894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5724.141995] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5724.167871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5724.180190] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5724.191137] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5724.228543] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5724.235753] RAX: 0000000000005d8f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5724.247744] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5724.258047] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5724.273819] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5724.276269] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5724.305587] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5724.313968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5724.326568] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5724.341911] Call Trace: [ 5724.345948] [ 5724.349268] ? show_regs.cold.9+0x22/0x2f [ 5724.359543] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5724.372286] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5724.373788] ? watchdog+0x40/0x40 [ 5724.374926] ? hrtimer_interrupt+0x128/0x2c0 [ 5724.378980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5724.383775] ? apic_timer_interrupt+0xf/0x20 [ 5724.386631] [ 5724.387790] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5724.410300] do_raw_spin_lock+0x177/0x190 [ 5724.420770] _raw_spin_lock+0x21/0x30 [ 5724.430071] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5724.438673] ? do_raw_spin_unlock+0x75/0x190 [ 5724.440274] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5724.456391] ? do_wait_intr+0xf0/0xf0 [ 5724.466080] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5724.467978] mdc_close+0x53b/0xd70 [mdc] [ 5724.476398] lmv_close+0x18d/0x490 [lmv] [ 5724.482093] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5724.491955] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5724.498643] ll_md_close+0x274/0x800 [lustre] [ 5724.506961] ll_file_release+0x128/0xc50 [lustre] [ 5724.518654] __fput+0x10d/0x440 [ 5724.519823] ____fput+0x12/0x20 [ 5724.529661] task_work_run+0xc6/0x110 [ 5724.532983] exit_to_usermode_loop+0x1dd/0x1f0 [ 5724.540312] do_syscall_64+0x3d6/0x440 [ 5724.552038] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5724.560667] RIP: 0033:0x7f1a63983cd8 [ 5724.571522] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5724.594753] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5724.602115] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5724.604372] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5724.627657] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5724.645642] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5724.653986] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5736.056119] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 5736.058005] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 5736.060867] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [ 5736.066090] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [ 5736.076726] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5736.076775] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5736.076783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5736.076786] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5736.076800] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5736.076804] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5736.076808] RAX: 0000000000005285 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5736.076811] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5736.076813] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5736.076815] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5736.076818] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5736.076821] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5736.076823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5736.076826] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5736.076834] Call Trace: [ 5736.076840] [ 5736.076845] ? show_regs.cold.9+0x22/0x2f [ 5736.076855] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5736.076861] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5736.076866] ? watchdog+0x40/0x40 [ 5736.076870] ? hrtimer_interrupt+0x128/0x2c0 [ 5736.076874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5736.076882] ? apic_timer_interrupt+0xf/0x20 [ 5736.076886] [ 5736.076889] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5736.076894] do_raw_spin_lock+0x177/0x190 [ 5736.076899] _raw_spin_lock+0x21/0x30 [ 5736.076972] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5736.086407] mdt(O) [ 5736.108844] ? do_raw_spin_unlock+0x75/0x190 [ 5736.125461] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5736.125726] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5736.125734] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5736.125741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5736.125756] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5736.125761] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5736.125766] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5736.125771] RDX: 0000000000003332 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5736.125775] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5736.125780] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5736.125783] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5736.125789] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5736.125792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5736.125795] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5736.125802] Call Trace: [ 5736.125808] [ 5736.125814] ? show_regs.cold.9+0x22/0x2f [ 5736.125824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5736.125830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5736.125834] ? watchdog+0x40/0x40 [ 5736.125838] ? hrtimer_interrupt+0x128/0x2c0 [ 5736.125843] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5736.125860] ? apic_timer_interrupt+0xf/0x20 [ 5736.125865] [ 5736.125868] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5736.125878] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5736.125883] do_raw_spin_lock+0x177/0x190 [ 5736.125888] _raw_spin_lock+0x21/0x30 [ 5736.125893] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5736.126634] obd_mod_free+0x46/0x140 [ptlrpc] [ 5736.127094] mdc_commit_open+0x91/0xb0 [mdc] [ 5736.132129] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5736.132786] ? do_wait_intr+0xf0/0xf0 [ 5736.132802] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5736.133235] mdc_close+0x53b/0xd70 [mdc] [ 5736.133274] lmv_close+0x18d/0x490 [lmv] [ 5736.133386] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5736.133531] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5736.133774] ll_md_close+0x274/0x800 [lustre] [ 5736.133860] ll_file_release+0x128/0xc50 [lustre] [ 5736.134039] __fput+0x10d/0x440 [ 5736.134049] ____fput+0x12/0x20 [ 5736.134054] task_work_run+0xc6/0x110 [ 5736.134062] exit_to_usermode_loop+0x1dd/0x1f0 [ 5736.134069] do_syscall_64+0x3d6/0x440 [ 5736.134074] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5736.134133] RIP: 0033:0x7f1a63983cd8 [ 5736.134141] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5736.134145] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5736.134150] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5736.134153] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5736.134155] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5736.134158] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5736.134160] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5736.373078] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5736.375697] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5736.378267] after_reply+0xe2c/0x1780 [ptlrpc] [ 5736.380094] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5736.382031] ? __next_timer_interrupt+0x160/0x160 [ 5736.383643] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5736.385234] ? do_wait_intr+0xf0/0xf0 [ 5736.387170] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5736.391748] kthread+0x1d1/0x200 [ 5736.395600] ? set_kthread_struct+0x70/0x70 [ 5736.397872] ret_from_fork+0x1f/0x30 [ 5744.053796] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5744.061273] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5744.097165] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5744.110503] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5744.125163] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5744.135648] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5744.155069] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5744.161315] RAX: 0000000000007d0a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5744.169357] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5744.180691] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5744.186751] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5744.190876] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5744.197160] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5744.202167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5744.207912] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5744.215833] Call Trace: [ 5744.218805] [ 5744.219817] ? show_regs.cold.9+0x22/0x2f [ 5744.225269] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5744.227163] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5744.231661] ? watchdog+0x40/0x40 [ 5744.234890] ? hrtimer_interrupt+0x128/0x2c0 [ 5744.242170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5744.244915] ? apic_timer_interrupt+0xf/0x20 [ 5744.248601] [ 5744.249303] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5744.254034] do_raw_spin_lock+0x177/0x190 [ 5744.260999] _raw_spin_lock+0x21/0x30 [ 5744.265562] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5744.272137] ? do_raw_spin_unlock+0x75/0x190 [ 5744.275791] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5744.282856] ? do_wait_intr+0xf0/0xf0 [ 5744.289068] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5744.294783] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5744.302719] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5744.312851] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5744.322096] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5744.330803] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5744.337034] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5744.338713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5744.354124] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5744.367075] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5744.373154] lmv_intent_open+0x338/0x1150 [lmv] [ 5744.376696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5744.383850] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5744.391190] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5744.394014] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5744.396770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5744.401038] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5744.403327] ? do_raw_spin_unlock+0x75/0x190 [ 5744.407964] ? _raw_spin_unlock+0x12/0x30 [ 5744.409863] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5744.415675] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5744.418980] ll_atomic_open+0x36a/0xd70 [lustre] [ 5744.424065] lookup_open+0x356/0xb40 [ 5744.426841] do_last+0x313/0xfc0 [ 5744.429740] ? nd_jump_root+0xe5/0x160 [ 5744.431604] ? path_init+0x437/0x520 [ 5744.432851] path_openat+0xf7/0x500 [ 5744.433989] ? do_raw_spin_unlock+0x75/0x190 [ 5744.445150] do_filp_open+0x99/0x140 [ 5744.447747] ? getname_flags+0x6e/0x330 [ 5744.450022] ? __check_object_size+0xff/0x256 [ 5744.452107] ? do_raw_spin_unlock+0x75/0x190 [ 5744.454018] ? _raw_spin_unlock+0x12/0x30 [ 5744.457973] do_sys_openat2+0x2b4/0x410 [ 5744.461744] do_sys_open+0x73/0xa0 [ 5744.462935] __x64_sys_openat+0x24/0x30 [ 5744.466223] do_syscall_64+0xc1/0x440 [ 5744.468192] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5744.470733] RIP: 0033:0x7f1a63983332 [ 5744.472899] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5744.499208] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5744.514006] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5744.520682] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5744.527513] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5744.533852] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5744.540829] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5747.175635] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3724s! [ 5747.182775] Showing busy workqueues and worker pools: [ 5747.185111] workqueue events: flags=0x0 [ 5747.190626] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5747.204800] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5747.207022] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5747.224930] pending: sec_gc_main [ptlrpc] [ 5747.235824] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5747.246023] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5747.255954] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5747.266241] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5747.277682] workqueue events_long: flags=0x0 [ 5747.282810] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5747.289067] pending: lu_objects_flush [obdclass] [ 5747.292581] workqueue events_unbound: flags=0x2 [ 5747.299028] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5747.310919] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5747.320026] workqueue events_power_efficient: flags=0x80 [ 5747.333145] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5747.336863] pending: neigh_periodic_work, do_cache_clean [ 5747.343059] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5747.358162] pending: neigh_periodic_work, check_lifetime [ 5747.365153] workqueue mm_percpu_wq: flags=0x8 [ 5747.372941] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5747.374614] pending: vmstat_update [ 5747.377854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5747.380606] pending: vmstat_update [ 5747.387928] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5747.396786] pending: vmstat_update [ 5747.400950] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5747.406344] pending: vmstat_update [ 5747.408483] workqueue writeback: flags=0x4a [ 5747.410317] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5747.412832] pending: wb_workfn, wb_workfn [ 5747.415115] workqueue xprtiod: flags=0xa [ 5747.417142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5747.421091] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5747.425457] workqueue ipv6_addrconf: flags=0x40008 [ 5747.427748] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5747.430839] pending: addrconf_verify_work [ 5747.433679] workqueue knbd0-recv: flags=0x1a [ 5747.438735] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5747.441352] in-flight: 647:recv_work [ 5747.445442] workqueue ptlrpc_pinger: flags=0x2 [ 5747.454289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5747.457355] pending: ptlrpc_pinger_main [ptlrpc] [ 5747.467047] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5737s workers=2 idle: 652 [ 5752.059452] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 5752.064721] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5752.101365] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5752.123949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5752.127168] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5752.130958] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5752.150894] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5752.157791] RAX: 0000000000000b7e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5752.165691] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5752.169062] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5752.185305] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5752.197545] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5752.210583] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5752.214892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5752.222492] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5752.226652] Call Trace: [ 5752.227863] [ 5752.229518] ? show_regs.cold.9+0x22/0x2f [ 5752.231715] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5752.233878] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5752.236740] ? watchdog+0x40/0x40 [ 5752.238387] ? hrtimer_interrupt+0x128/0x2c0 [ 5752.240838] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5752.242972] ? apic_timer_interrupt+0xf/0x20 [ 5752.244954] [ 5752.245918] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5752.248760] do_raw_spin_lock+0x177/0x190 [ 5752.251837] _raw_spin_lock+0x21/0x30 [ 5752.254966] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5752.259575] ? do_raw_spin_unlock+0x75/0x190 [ 5752.260917] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5752.263881] ? do_wait_intr+0xf0/0xf0 [ 5752.265610] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5752.268972] mdc_close+0x53b/0xd70 [mdc] [ 5752.271757] lmv_close+0x18d/0x490 [lmv] [ 5752.276812] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5752.289066] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5752.290999] ll_md_close+0x274/0x800 [lustre] [ 5752.300631] ll_file_release+0x128/0xc50 [lustre] [ 5752.304688] __fput+0x10d/0x440 [ 5752.310913] ____fput+0x12/0x20 [ 5752.313552] task_work_run+0xc6/0x110 [ 5752.316642] exit_to_usermode_loop+0x1dd/0x1f0 [ 5752.318121] do_syscall_64+0x3d6/0x440 [ 5752.319977] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5752.321836] RIP: 0033:0x7f1a63983cd8 [ 5752.323564] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5752.331873] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5752.335089] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5752.338553] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5752.341674] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5752.350327] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5752.354835] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5764.056432] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 5764.059975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5764.060800] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 5764.107243] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5764.110494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5764.110567] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5764.110574] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5764.110576] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5764.110591] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5764.110595] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5764.110600] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5764.110602] RDX: 0000000000007db3 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5764.110605] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5764.110607] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5764.110609] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5764.110612] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5764.110615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5764.110618] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5764.110628] Call Trace: [ 5764.110633] [ 5764.110639] ? show_regs.cold.9+0x22/0x2f [ 5764.110725] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5764.110732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5764.110736] ? watchdog+0x40/0x40 [ 5764.110740] ? hrtimer_interrupt+0x128/0x2c0 [ 5764.110745] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5764.110752] ? apic_timer_interrupt+0xf/0x20 [ 5764.110757] [ 5764.110760] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5764.110765] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5764.110769] do_raw_spin_lock+0x177/0x190 [ 5764.110774] _raw_spin_lock+0x21/0x30 [ 5764.110779] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5764.111061] obd_mod_free+0x46/0x140 [ptlrpc] [ 5764.111262] mdc_commit_open+0x91/0xb0 [mdc] [ 5764.111296] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5764.111487] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5764.111822] after_reply+0xe2c/0x1780 [ptlrpc] [ 5764.112060] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5764.112260] ? __next_timer_interrupt+0x160/0x160 [ 5764.112270] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5764.112470] ? do_wait_intr+0xf0/0xf0 [ 5764.112480] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5764.112757] kthread+0x1d1/0x200 [ 5764.112764] ? set_kthread_struct+0x70/0x70 [ 5764.112768] ret_from_fork+0x1f/0x30 [ 5764.327205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5764.329605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5764.331261] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5764.336851] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5764.340142] RAX: 00000000000007e3 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5764.342928] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5764.344653] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5764.346409] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5764.348524] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5764.350246] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5764.355234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5764.357907] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5764.359913] Call Trace: [ 5764.360674] [ 5764.361306] ? show_regs.cold.9+0x22/0x2f [ 5764.362601] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5764.364956] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5764.368834] ? watchdog+0x40/0x40 [ 5764.371088] ? hrtimer_interrupt+0x128/0x2c0 [ 5764.373918] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5764.378445] ? apic_timer_interrupt+0xf/0x20 [ 5764.380691] [ 5764.381760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5764.384444] do_raw_spin_lock+0x177/0x190 [ 5764.388872] _raw_spin_lock+0x21/0x30 [ 5764.390840] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5764.395785] ? do_raw_spin_unlock+0x75/0x190 [ 5764.397795] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5764.402698] ? do_wait_intr+0xf0/0xf0 [ 5764.404537] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5764.406857] mdc_close+0x53b/0xd70 [mdc] [ 5764.408920] lmv_close+0x18d/0x490 [lmv] [ 5764.410632] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5764.413153] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5764.415561] ll_md_close+0x274/0x800 [lustre] [ 5764.418500] ll_file_release+0x128/0xc50 [lustre] [ 5764.421333] __fput+0x10d/0x440 [ 5764.423681] ____fput+0x12/0x20 [ 5764.425156] task_work_run+0xc6/0x110 [ 5764.427322] exit_to_usermode_loop+0x1dd/0x1f0 [ 5764.429423] do_syscall_64+0x3d6/0x440 [ 5764.431748] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5764.434145] RIP: 0033:0x7f1a63983cd8 [ 5764.435704] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5764.443809] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5764.446796] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5764.449843] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5764.453418] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5764.456269] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5764.459205] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5772.053507] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5772.055821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5772.078441] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5772.081925] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5772.085723] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5772.087257] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5772.096021] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5772.098243] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5772.100517] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5772.102405] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5772.104607] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5772.106453] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5772.108312] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5772.110322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5772.111880] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5772.113586] Call Trace: [ 5772.114284] [ 5772.114823] ? show_regs.cold.9+0x22/0x2f [ 5772.115906] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5772.118108] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5772.120216] ? watchdog+0x40/0x40 [ 5772.122148] ? hrtimer_interrupt+0x128/0x2c0 [ 5772.132050] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5772.136684] ? apic_timer_interrupt+0xf/0x20 [ 5772.140679] [ 5772.141645] ? native_safe_halt+0x2e/0x50 [ 5772.146995] kvm_wait+0xcd/0xe0 [ 5772.149284] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5772.154426] do_raw_spin_lock+0x177/0x190 [ 5772.156039] _raw_spin_lock+0x21/0x30 [ 5772.159094] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5772.161724] ? do_raw_spin_unlock+0x75/0x190 [ 5772.163608] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5772.165536] ? do_wait_intr+0xf0/0xf0 [ 5772.166976] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5772.169644] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5772.172482] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5772.175203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5772.181047] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5772.182681] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5772.186861] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5772.190460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5772.194557] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5772.197931] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5772.200343] lmv_intent_open+0x338/0x1150 [lmv] [ 5772.204712] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5772.208478] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5772.217595] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5772.222120] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5772.225607] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5772.229303] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5772.232260] ? do_raw_spin_unlock+0x75/0x190 [ 5772.235159] ? _raw_spin_unlock+0x12/0x30 [ 5772.238236] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5772.244574] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5772.247819] ll_atomic_open+0x36a/0xd70 [lustre] [ 5772.253387] lookup_open+0x356/0xb40 [ 5772.257332] do_last+0x313/0xfc0 [ 5772.261904] ? nd_jump_root+0xe5/0x160 [ 5772.264267] ? path_init+0x437/0x520 [ 5772.269057] path_openat+0xf7/0x500 [ 5772.272257] ? do_raw_spin_unlock+0x75/0x190 [ 5772.278234] do_filp_open+0x99/0x140 [ 5772.281158] ? getname_flags+0x6e/0x330 [ 5772.282538] ? __check_object_size+0xff/0x256 [ 5772.285197] ? do_raw_spin_unlock+0x75/0x190 [ 5772.287435] ? _raw_spin_unlock+0x12/0x30 [ 5772.289441] do_sys_openat2+0x2b4/0x410 [ 5772.293273] do_sys_open+0x73/0xa0 [ 5772.298439] __x64_sys_openat+0x24/0x30 [ 5772.299860] do_syscall_64+0xc1/0x440 [ 5772.301375] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5772.302892] RIP: 0033:0x7f1a63983332 [ 5772.305344] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5772.314182] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5772.317356] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5772.323259] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5772.326855] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5772.332574] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5772.336144] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5777.889041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3754s! [ 5777.892834] Showing busy workqueues and worker pools: [ 5777.896083] workqueue events: flags=0x0 [ 5777.902116] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5777.905517] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5777.910587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5777.914333] pending: sec_gc_main [ptlrpc] [ 5777.917367] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5777.921329] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5777.926558] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5777.932018] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5777.937736] workqueue events_long: flags=0x0 [ 5777.940317] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5777.944027] pending: lu_objects_flush [obdclass] [ 5777.947315] workqueue events_unbound: flags=0x2 [ 5777.949904] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5777.953404] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5777.957954] workqueue events_power_efficient: flags=0x80 [ 5777.961187] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5777.964548] pending: neigh_periodic_work, do_cache_clean [ 5777.966771] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5777.970505] pending: neigh_periodic_work, check_lifetime [ 5777.973638] workqueue mm_percpu_wq: flags=0x8 [ 5777.975907] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5777.979550] pending: vmstat_update [ 5777.981969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5777.985591] pending: vmstat_update [ 5777.987949] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5777.991448] pending: vmstat_update [ 5777.993805] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5778.000113] pending: vmstat_update [ 5778.002563] workqueue writeback: flags=0x4a [ 5778.005074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5778.008121] pending: wb_workfn, wb_workfn [ 5778.011395] workqueue xprtiod: flags=0xa [ 5778.013286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5778.018880] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5778.022947] workqueue ipv6_addrconf: flags=0x40008 [ 5778.026130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5778.029639] pending: addrconf_verify_work [ 5778.032123] workqueue knbd0-recv: flags=0x1a [ 5778.034866] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5778.038581] in-flight: 647:recv_work [ 5778.040739] workqueue ptlrpc_pinger: flags=0x2 [ 5778.043406] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5778.046682] pending: ptlrpc_pinger_main [ptlrpc] [ 5778.049851] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5768s workers=2 idle: 652 [ 5780.057043] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 5780.062848] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5780.101805] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5780.108599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5780.113621] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5780.118824] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5780.131733] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5780.135096] RAX: 0000000000000c97 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5780.138751] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5780.142347] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5780.144455] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5780.148852] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5780.152625] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5780.156737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5780.158437] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5780.160811] Call Trace: [ 5780.162024] [ 5780.162795] ? show_regs.cold.9+0x22/0x2f [ 5780.164434] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5780.166474] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5780.168005] ? watchdog+0x40/0x40 [ 5780.170132] ? hrtimer_interrupt+0x128/0x2c0 [ 5780.171231] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5780.172787] ? apic_timer_interrupt+0xf/0x20 [ 5780.183457] [ 5780.189758] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5780.193637] do_raw_spin_lock+0x177/0x190 [ 5780.195288] _raw_spin_lock+0x21/0x30 [ 5780.196431] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5780.198317] ? do_raw_spin_unlock+0x75/0x190 [ 5780.200009] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5780.202118] ? do_wait_intr+0xf0/0xf0 [ 5780.203516] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5780.205700] mdc_close+0x53b/0xd70 [mdc] [ 5780.207372] lmv_close+0x18d/0x490 [lmv] [ 5780.208991] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5780.211646] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5780.213663] ll_md_close+0x274/0x800 [lustre] [ 5780.215528] ll_file_release+0x128/0xc50 [lustre] [ 5780.217531] __fput+0x10d/0x440 [ 5780.218804] ____fput+0x12/0x20 [ 5780.219794] task_work_run+0xc6/0x110 [ 5780.221345] exit_to_usermode_loop+0x1dd/0x1f0 [ 5780.222687] do_syscall_64+0x3d6/0x440 [ 5780.224201] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5780.226472] RIP: 0033:0x7f1a63983cd8 [ 5780.228221] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5780.235970] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5780.245340] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5780.248210] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5780.251983] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5780.255040] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5780.258039] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5792.058616] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 5792.060476] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 5792.065103] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5792.065191] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5792.065200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5792.065203] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5792.065216] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5792.065221] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5792.065347] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5792.065350] RDX: 00000000000064ac RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5792.065353] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5792.065355] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5792.065357] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5792.065361] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5792.065364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5792.065366] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5792.065382] Call Trace: [ 5792.065388] [ 5792.065394] ? show_regs.cold.9+0x22/0x2f [ 5792.065404] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5792.065412] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5792.065417] ? watchdog+0x40/0x40 [ 5792.065421] ? hrtimer_interrupt+0x128/0x2c0 [ 5792.065425] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5792.065437] ? apic_timer_interrupt+0xf/0x20 [ 5792.065442] [ 5792.065445] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5792.065449] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5792.065455] do_raw_spin_lock+0x177/0x190 [ 5792.065460] _raw_spin_lock+0x21/0x30 [ 5792.065465] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5792.066102] obd_mod_free+0x46/0x140 [ptlrpc] [ 5792.066535] mdc_commit_open+0x91/0xb0 [mdc] [ 5792.066575] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5792.074696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [ 5792.114702] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5792.115177] after_reply+0xe2c/0x1780 [ptlrpc] [ 5792.116141] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5792.116439] ? __next_timer_interrupt+0x160/0x160 [ 5792.116451] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5792.117991] ? do_wait_intr+0xf0/0xf0 [ 5792.118003] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5792.119051] kthread+0x1d1/0x200 [ 5792.119065] ? set_kthread_struct+0x70/0x70 [ 5792.119070] ret_from_fork+0x1f/0x30 [ 5792.387436] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5792.400410] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5792.404495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5792.408048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5792.410336] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5792.417693] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5792.420576] RAX: 000000000000520a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5792.423485] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5792.426413] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5792.429327] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5792.435401] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5792.446860] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5792.453368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5792.461299] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5792.464149] Call Trace: [ 5792.467134] [ 5792.469219] ? show_regs.cold.9+0x22/0x2f [ 5792.473422] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5792.478600] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5792.483057] ? watchdog+0x40/0x40 [ 5792.486004] ? hrtimer_interrupt+0x128/0x2c0 [ 5792.491120] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5792.498289] ? apic_timer_interrupt+0xf/0x20 [ 5792.500556] [ 5792.501886] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5792.512153] do_raw_spin_lock+0x177/0x190 [ 5792.519994] _raw_spin_lock+0x21/0x30 [ 5792.523020] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5792.535769] ? do_raw_spin_unlock+0x75/0x190 [ 5792.542334] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5792.551986] ? do_wait_intr+0xf0/0xf0 [ 5792.558423] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5792.566898] mdc_close+0x53b/0xd70 [mdc] [ 5792.572939] lmv_close+0x18d/0x490 [lmv] [ 5792.577558] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5792.584337] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5792.588666] ll_md_close+0x274/0x800 [lustre] [ 5792.594012] ll_file_release+0x128/0xc50 [lustre] [ 5792.602086] __fput+0x10d/0x440 [ 5792.611397] ____fput+0x12/0x20 [ 5792.613056] task_work_run+0xc6/0x110 [ 5792.622087] exit_to_usermode_loop+0x1dd/0x1f0 [ 5792.624257] do_syscall_64+0x3d6/0x440 [ 5792.631410] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5792.633652] RIP: 0033:0x7f1a63983cd8 [ 5792.644073] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5792.662589] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5792.670304] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5792.679278] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5792.688695] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5792.699160] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5792.708649] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5804.053066] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5804.055790] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5804.082404] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5804.090880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5804.095352] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5804.098381] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5804.107749] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5804.112191] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5804.116782] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5804.123252] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5804.128135] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5804.133271] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5804.145783] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5804.157343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5804.169608] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5804.177010] Call Trace: [ 5804.179771] [ 5804.181351] ? show_regs.cold.9+0x22/0x2f [ 5804.182955] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5804.187550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5804.191610] ? watchdog+0x40/0x40 [ 5804.196333] ? hrtimer_interrupt+0x128/0x2c0 [ 5804.198968] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5804.202012] ? apic_timer_interrupt+0xf/0x20 [ 5804.203458] [ 5804.205002] ? native_safe_halt+0x2e/0x50 [ 5804.207059] kvm_wait+0xcd/0xe0 [ 5804.210706] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5804.217757] do_raw_spin_lock+0x177/0x190 [ 5804.228202] _raw_spin_lock+0x21/0x30 [ 5804.235270] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5804.244036] ? do_raw_spin_unlock+0x75/0x190 [ 5804.256064] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5804.258129] ? do_wait_intr+0xf0/0xf0 [ 5804.271091] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5804.280711] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5804.288007] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5804.292437] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5804.295429] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5804.307729] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5804.314550] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5804.316779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5804.325326] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5804.332856] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5804.337177] lmv_intent_open+0x338/0x1150 [lmv] [ 5804.342767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5804.347774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5804.350917] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5804.352048] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5804.353500] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5804.362539] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5804.366184] ? do_raw_spin_unlock+0x75/0x190 [ 5804.369335] ? _raw_spin_unlock+0x12/0x30 [ 5804.373232] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5804.381018] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5804.384231] ll_atomic_open+0x36a/0xd70 [lustre] [ 5804.387735] lookup_open+0x356/0xb40 [ 5804.389794] do_last+0x313/0xfc0 [ 5804.390918] ? nd_jump_root+0xe5/0x160 [ 5804.397028] ? path_init+0x437/0x520 [ 5804.398402] path_openat+0xf7/0x500 [ 5804.403908] ? do_raw_spin_unlock+0x75/0x190 [ 5804.406600] do_filp_open+0x99/0x140 [ 5804.414061] ? getname_flags+0x6e/0x330 [ 5804.415957] ? __check_object_size+0xff/0x256 [ 5804.418213] ? do_raw_spin_unlock+0x75/0x190 [ 5804.420432] ? _raw_spin_unlock+0x12/0x30 [ 5804.424793] do_sys_openat2+0x2b4/0x410 [ 5804.426330] do_sys_open+0x73/0xa0 [ 5804.428894] __x64_sys_openat+0x24/0x30 [ 5804.430455] do_syscall_64+0xc1/0x440 [ 5804.434241] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5804.439704] RIP: 0033:0x7f1a63983332 [ 5804.445297] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5804.462779] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5804.471455] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5804.476876] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5804.481597] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5804.484809] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5804.488371] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5808.057893] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 5808.061438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5808.099028] BUG: workqueue lockup - pool [ 5808.112470] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5808.112496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5808.112499] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5808.112514] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5808.112518] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5808.112529] RAX: 0000000000004514 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5808.112532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5808.112534] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5808.112536] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5808.112538] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5808.112541] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5808.112544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5808.112546] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5808.112555] Call Trace: [ 5808.112559] [ 5808.112564] ? show_regs.cold.9+0x22/0x2f [ 5808.112574] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5808.112580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5808.112591] ? watchdog+0x40/0x40 [ 5808.112605] ? hrtimer_interrupt+0x128/0x2c0 [ 5808.112609] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5808.112616] ? apic_timer_interrupt+0xf/0x20 [ 5808.112620] [ 5808.112623] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5808.112629] do_raw_spin_lock+0x177/0x190 [ 5808.112633] _raw_spin_lock+0x21/0x30 [ 5808.112638] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5808.112925] ? do_raw_spin_unlock+0x75/0x190 [ 5808.113072] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5808.115389] ? do_wait_intr+0xf0/0xf0 [ 5808.115400] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5808.122226] cpus=0-3 flags=0x4 nice=0 stuck for 3785s! [ 5808.122249] Showing busy workqueues and worker pools: [ 5808.122252] workqueue events: flags=0x0 [ 5808.122256] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5808.122275] pending: ldlm_pools_recalc_task [ptlrpc] [ 5808.139991] mdc_close+0x53b/0xd70 [mdc] [ 5808.140043] lmv_close+0x18d/0x490 [lmv] [ 5808.141126] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5808.141232] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5808.141316] ll_md_close+0x274/0x800 [lustre] [ 5808.141403] ll_file_release+0x128/0xc50 [lustre] [ 5808.141492] __fput+0x10d/0x440 [ 5808.141501] ____fput+0x12/0x20 [ 5808.141506] task_work_run+0xc6/0x110 [ 5808.141513] exit_to_usermode_loop+0x1dd/0x1f0 [ 5808.141522] do_syscall_64+0x3d6/0x440 [ 5808.141540] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5808.141549] RIP: 0033:0x7f1a63983cd8 [ 5808.141556] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5808.141559] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5808.141564] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5808.141567] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5808.141569] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5808.141571] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5808.141573] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5808.580961] , push_to_pool [ 5808.582659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5808.585716] pending: sec_gc_main [ptlrpc] [ 5808.588121] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5808.591245] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5808.595903] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5808.599135] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5808.604296] workqueue events_long: flags=0x0 [ 5808.607352] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5808.611622] pending: lu_objects_flush [obdclass] [ 5808.620377] workqueue events_unbound: flags=0x2 [ 5808.625915] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5808.633236] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5808.642330] workqueue events_power_efficient: flags=0x80 [ 5808.648424] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5808.655572] pending: neigh_periodic_work, do_cache_clean [ 5808.666063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5808.671782] pending: neigh_periodic_work, check_lifetime [ 5808.676357] workqueue mm_percpu_wq: flags=0x8 [ 5808.682427] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5808.688923] pending: vmstat_update [ 5808.690769] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5808.694202] pending: vmstat_update [ 5808.701881] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5808.707365] pending: vmstat_update [ 5808.710070] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5808.714031] pending: vmstat_update [ 5808.716076] workqueue writeback: flags=0x4a [ 5808.717900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5808.720772] pending: wb_workfn, wb_workfn [ 5808.723195] workqueue xprtiod: flags=0xa [ 5808.728903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5808.735516] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5808.747458] workqueue ipv6_addrconf: flags=0x40008 [ 5808.750785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5808.754743] pending: addrconf_verify_work [ 5808.758026] workqueue knbd0-recv: flags=0x1a [ 5808.765511] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5808.777516] in-flight: 647:recv_work [ 5808.779847] workqueue ptlrpc_pinger: flags=0x2 [ 5808.783858] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5808.795544] pending: ptlrpc_pinger_main [ptlrpc] [ 5808.799772] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5798s workers=2 idle: 652 [ 5820.063359] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5820.063415] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5820.070456] Modules linked in: lustre(O) [ 5820.079162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5820.079251] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5820.079259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5820.079262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5820.079275] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5820.079279] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5820.079284] RAX: 0000000000001082 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5820.079287] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5820.079289] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5820.079291] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5820.079294] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5820.079297] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5820.079300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5820.079302] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5820.079312] Call Trace: [ 5820.079318] [ 5820.079324] ? show_regs.cold.9+0x22/0x2f [ 5820.079336] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5820.079343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5820.079348] ? watchdog+0x40/0x40 [ 5820.079352] ? hrtimer_interrupt+0x128/0x2c0 [ 5820.079356] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5820.079364] ? apic_timer_interrupt+0xf/0x20 [ 5820.079368] [ 5820.079370] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5820.079377] do_raw_spin_lock+0x177/0x190 [ 5820.079382] _raw_spin_lock+0x21/0x30 [ 5820.079387] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5820.079909] ? do_raw_spin_unlock+0x75/0x190 [ 5820.079915] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5820.080280] ? do_wait_intr+0xf0/0xf0 [ 5820.080288] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5820.082246] mdc_close+0x53b/0xd70 [mdc] [ 5820.082295] lmv_close+0x18d/0x490 [lmv] [ 5820.082325] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5820.091965] osp(O) [ 5820.133569] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5820.133697] ll_md_close+0x274/0x800 [lustre] [ 5820.133782] ll_file_release+0x128/0xc50 [lustre] [ 5820.133862] __fput+0x10d/0x440 [ 5820.134024] ____fput+0x12/0x20 [ 5820.134029] task_work_run+0xc6/0x110 [ 5820.134036] exit_to_usermode_loop+0x1dd/0x1f0 [ 5820.134042] do_syscall_64+0x3d6/0x440 [ 5820.134047] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5820.134055] RIP: 0033:0x7f1a63983cd8 [ 5820.134062] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5820.134066] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5820.134071] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5820.134074] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5820.134076] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5820.134078] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5820.134080] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5820.576687] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5820.625814] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5820.644069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5820.655160] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5820.660426] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5820.698963] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5820.713738] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5820.718208] RDX: 00000000000032ea RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5820.728058] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5820.740793] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5820.743305] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5820.757813] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5820.761380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5820.766280] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5820.778776] Call Trace: [ 5820.783169] [ 5820.785618] ? show_regs.cold.9+0x22/0x2f [ 5820.793478] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5820.795172] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5820.806940] ? watchdog+0x40/0x40 [ 5820.812850] ? hrtimer_interrupt+0x128/0x2c0 [ 5820.816325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5820.822012] ? apic_timer_interrupt+0xf/0x20 [ 5820.828187] [ 5820.833870] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5820.840041] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5820.849142] do_raw_spin_lock+0x177/0x190 [ 5820.854638] _raw_spin_lock+0x21/0x30 [ 5820.856269] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5820.864667] obd_mod_free+0x46/0x140 [ptlrpc] [ 5820.867249] mdc_commit_open+0x91/0xb0 [mdc] [ 5820.870992] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5820.879239] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5820.883370] after_reply+0xe2c/0x1780 [ptlrpc] [ 5820.891360] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5820.902446] ? __next_timer_interrupt+0x160/0x160 [ 5820.904099] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5820.911308] ? do_wait_intr+0xf0/0xf0 [ 5820.915866] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5820.926214] kthread+0x1d1/0x200 [ 5820.927456] ? set_kthread_struct+0x70/0x70 [ 5820.938779] ret_from_fork+0x1f/0x30 [ 5836.054894] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 5836.059150] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 5836.059612] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5836.059690] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5836.059696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5836.059699] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5836.059714] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5836.059718] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5836.059722] RAX: 0000000000005e29 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5836.059725] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5836.059728] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5836.059730] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5836.059732] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5836.059735] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5836.059738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5836.059740] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5836.059748] Call Trace: [ 5836.059754] [ 5836.059759] ? show_regs.cold.9+0x22/0x2f [ 5836.059769] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5836.059775] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5836.059780] ? watchdog+0x40/0x40 [ 5836.059785] ? hrtimer_interrupt+0x128/0x2c0 [ 5836.059789] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5836.059981] ? apic_timer_interrupt+0xf/0x20 [ 5836.059989] [ 5836.059997] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5836.060004] do_raw_spin_lock+0x177/0x190 [ 5836.060009] _raw_spin_lock+0x21/0x30 [ 5836.060013] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5836.060466] ? do_raw_spin_unlock+0x75/0x190 [ 5836.060473] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5836.060758] ? do_wait_intr+0xf0/0xf0 [ 5836.060767] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5836.061119] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5836.061402] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5836.061761] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5836.061959] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5836.061997] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5836.062220] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5836.062247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5836.072456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5836.072537] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5836.072544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5836.072547] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5836.072560] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5836.072720] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5836.072725] RAX: 000000000000718e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5836.072727] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5836.072729] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5836.072731] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5836.072733] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5836.072737] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5836.072741] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5836.072743] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5836.072752] Call Trace: [ 5836.072757] [ 5836.072763] ? show_regs.cold.9+0x22/0x2f [ 5836.072772] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5836.072778] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5836.072782] ? watchdog+0x40/0x40 [ 5836.072786] ? hrtimer_interrupt+0x128/0x2c0 [ 5836.072789] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5836.072797] ? apic_timer_interrupt+0xf/0x20 [ 5836.072879] [ 5836.072883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5836.072889] do_raw_spin_lock+0x177/0x190 [ 5836.072893] _raw_spin_lock+0x21/0x30 [ 5836.072898] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5836.073514] ? do_raw_spin_unlock+0x75/0x190 [ 5836.073520] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5836.073953] ? do_wait_intr+0xf0/0xf0 [ 5836.073967] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5836.109353] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5836.117880] mdc_close+0x53b/0xd70 [mdc] [ 5836.117935] lmv_close+0x18d/0x490 [lmv] [ 5836.117969] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5836.134538] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5836.134651] lmv_intent_open+0x338/0x1150 [lmv] [ 5836.134684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5836.134811] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5836.135010] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5836.135103] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5836.135383] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5836.135468] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5836.135552] ? do_raw_spin_unlock+0x75/0x190 [ 5836.135560] ? _raw_spin_unlock+0x12/0x30 [ 5836.135566] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5836.135597] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5836.142576] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5836.142763] ll_md_close+0x274/0x800 [lustre] [ 5836.142971] ll_file_release+0x128/0xc50 [lustre] [ 5836.143067] __fput+0x10d/0x440 [ 5836.143078] ____fput+0x12/0x20 [ 5836.143083] task_work_run+0xc6/0x110 [ 5836.143092] exit_to_usermode_loop+0x1dd/0x1f0 [ 5836.146100] do_syscall_64+0x3d6/0x440 [ 5836.146109] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5836.146117] RIP: 0033:0x7f1a63983cd8 [ 5836.146126] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5836.146130] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5836.146135] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5836.146137] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5836.146139] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5836.146142] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5836.146144] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5836.701835] ll_atomic_open+0x36a/0xd70 [lustre] [ 5836.708176] lookup_open+0x356/0xb40 [ 5836.709953] do_last+0x313/0xfc0 [ 5836.714952] ? nd_jump_root+0xe5/0x160 [ 5836.716848] ? path_init+0x437/0x520 [ 5836.723132] path_openat+0xf7/0x500 [ 5836.725122] ? do_raw_spin_unlock+0x75/0x190 [ 5836.730762] do_filp_open+0x99/0x140 [ 5836.733091] ? getname_flags+0x6e/0x330 [ 5836.736461] ? __check_object_size+0xff/0x256 [ 5836.741234] ? do_raw_spin_unlock+0x75/0x190 [ 5836.743887] ? _raw_spin_unlock+0x12/0x30 [ 5836.746333] do_sys_openat2+0x2b4/0x410 [ 5836.748152] do_sys_open+0x73/0xa0 [ 5836.750152] __x64_sys_openat+0x24/0x30 [ 5836.753949] do_syscall_64+0xc1/0x440 [ 5836.757689] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5836.761972] RIP: 0033:0x7f1a63983332 [ 5836.763219] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5836.784668] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5836.794501] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5836.804676] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5836.814493] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5836.819880] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5836.824960] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5838.816117] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3815s! [ 5838.818541] Showing busy workqueues and worker pools: [ 5838.825097] workqueue events: flags=0x0 [ 5838.828364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5838.835659] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5838.844469] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5838.851106] pending: sec_gc_main [ptlrpc] [ 5838.854154] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5838.857891] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5838.865080] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5838.871862] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5838.877263] workqueue events_long: flags=0x0 [ 5838.878378] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5838.884567] pending: lu_objects_flush [obdclass] [ 5838.894490] workqueue events_unbound: flags=0x2 [ 5838.896740] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5838.903903] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5838.910502] workqueue events_power_efficient: flags=0x80 [ 5838.912782] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5838.921679] pending: neigh_periodic_work, do_cache_clean [ 5838.926617] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5838.929550] pending: neigh_periodic_work, check_lifetime [ 5838.933395] workqueue mm_percpu_wq: flags=0x8 [ 5838.935379] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5838.941837] pending: vmstat_update [ 5838.945659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5838.952608] pending: vmstat_update [ 5838.955387] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5838.960544] pending: vmstat_update [ 5838.964948] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5838.976283] pending: vmstat_update [ 5838.981603] workqueue writeback: flags=0x4a [ 5838.986068] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5838.992702] pending: wb_workfn, wb_workfn [ 5838.994591] workqueue xprtiod: flags=0xa [ 5838.998740] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5839.001582] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5839.007123] workqueue ipv6_addrconf: flags=0x40008 [ 5839.012447] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5839.020094] pending: addrconf_verify_work [ 5839.022630] workqueue knbd0-recv: flags=0x1a [ 5839.025030] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5839.029111] in-flight: 647:recv_work [ 5839.038692] workqueue ptlrpc_pinger: flags=0x2 [ 5839.042735] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5839.048972] pending: ptlrpc_pinger_main [ptlrpc] [ 5839.057190] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5829s workers=2 idle: 652 [ 5848.056771] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5848.058682] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5848.062090] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5848.062175] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5848.062182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5848.062184] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5848.062719] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5848.067390] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5848.067463] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5848.067471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5848.067474] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5848.067488] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5848.067492] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5848.067496] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5848.067498] RDX: 000000000000221a RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5848.067501] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5848.067503] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5848.067505] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5848.067508] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5848.067511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5848.067513] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5848.067522] Call Trace: [ 5848.067528] [ 5848.067534] ? show_regs.cold.9+0x22/0x2f [ 5848.067544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5848.067551] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5848.067555] ? watchdog+0x40/0x40 [ 5848.067560] ? hrtimer_interrupt+0x128/0x2c0 [ 5848.067564] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5848.067573] ? apic_timer_interrupt+0xf/0x20 [ 5848.067578] [ 5848.067581] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5848.067585] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5848.067590] do_raw_spin_lock+0x177/0x190 [ 5848.067594] _raw_spin_lock+0x21/0x30 [ 5848.067599] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5848.067871] obd_mod_free+0x46/0x140 [ptlrpc] [ 5848.097584] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5848.097598] RAX: 0000000000007f78 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5848.097601] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5848.097603] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5848.097605] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5848.097607] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5848.097610] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5848.097612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5848.097615] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5848.097623] Call Trace: [ 5848.097629] [ 5848.097635] ? show_regs.cold.9+0x22/0x2f [ 5848.097648] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5848.097654] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5848.097657] ? watchdog+0x40/0x40 [ 5848.097661] ? hrtimer_interrupt+0x128/0x2c0 [ 5848.097665] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5848.097673] ? apic_timer_interrupt+0xf/0x20 [ 5848.097677] [ 5848.097679] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5848.097688] do_raw_spin_lock+0x177/0x190 [ 5848.097693] _raw_spin_lock+0x21/0x30 [ 5848.097697] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5848.098129] ? do_raw_spin_unlock+0x75/0x190 [ 5848.098136] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5848.098483] ? do_wait_intr+0xf0/0xf0 [ 5848.098490] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5848.106112] mdc_commit_open+0x91/0xb0 [mdc] [ 5848.106179] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5848.106597] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5848.106792] after_reply+0xe2c/0x1780 [ptlrpc] [ 5848.106986] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5848.107179] ? __next_timer_interrupt+0x160/0x160 [ 5848.107189] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5848.112963] mdc_close+0x53b/0xd70 [mdc] [ 5848.113026] lmv_close+0x18d/0x490 [lmv] [ 5848.113058] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5848.113191] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5848.113287] ll_md_close+0x274/0x800 [lustre] [ 5848.113511] ll_file_release+0x128/0xc50 [lustre] [ 5848.113593] __fput+0x10d/0x440 [ 5848.113602] ____fput+0x12/0x20 [ 5848.113608] task_work_run+0xc6/0x110 [ 5848.113615] exit_to_usermode_loop+0x1dd/0x1f0 [ 5848.113623] do_syscall_64+0x3d6/0x440 [ 5848.113628] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5848.113637] RIP: 0033:0x7f1a63983cd8 [ 5848.113644] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5848.113648] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5848.113652] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5848.113655] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5848.113656] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5848.113658] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5848.113661] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5848.469038] ? do_wait_intr+0xf0/0xf0 [ 5848.472873] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5848.480382] kthread+0x1d1/0x200 [ 5848.481270] ? set_kthread_struct+0x70/0x70 [ 5848.487337] ret_from_fork+0x1f/0x30 [ 5862.494015] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 5862.497015] rcu: 3-...!: (2347646 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 5862.500817] (detected by 1, t=3840088 jiffies, g=269633, q=1530) [ 5862.503208] Sending NMI from CPU 1 to CPUs 3: [ 5862.508889] NMI backtrace for cpu 3 [ 5862.508896] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5862.508898] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5862.508900] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5862.508902] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5862.508905] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 5862.508908] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5862.508909] RDX: 0000000000004803 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5862.508911] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5862.508913] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5862.508914] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5862.508916] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5862.508917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5862.508919] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5862.508920] Call Trace: [ 5862.508921] [ 5862.508922] ? show_regs.cold.9+0x22/0x2f [ 5862.508923] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5862.508924] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5862.508926] ? nmi_handle+0xbd/0x210 [ 5862.508927] ? default_do_nmi+0x4b/0x1b0 [ 5862.508928] ? do_nmi+0x280/0x380 [ 5862.508930] ? end_repeat_nmi+0x16/0x1f [ 5862.508931] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5862.508932] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5862.508934] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5862.508935] [ 5862.508936] do_raw_spin_lock+0x177/0x190 [ 5862.508937] _raw_spin_lock+0x21/0x30 [ 5862.508938] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5862.508939] obd_mod_free+0x46/0x140 [ptlrpc] [ 5862.508940] mdc_commit_open+0x91/0xb0 [mdc] [ 5862.508942] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5862.508943] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5862.508944] after_reply+0xe2c/0x1780 [ptlrpc] [ 5862.508945] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5862.508946] ? __next_timer_interrupt+0x160/0x160 [ 5862.508947] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5862.508948] ? do_wait_intr+0xf0/0xf0 [ 5862.508949] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5862.508950] kthread+0x1d1/0x200 [ 5862.508951] ? set_kthread_struct+0x70/0x70 [ 5862.508952] ret_from_fork+0x1f/0x30 [ 5862.509220] rcu: rcu_sched kthread starved for 3839734 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 5862.652520] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5862.655299] rcu: RCU grace-period kthread stack dump: [ 5862.656824] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5862.659428] Call Trace: [ 5862.660177] __schedule+0x351/0xcb0 [ 5862.661212] schedule+0xc0/0x180 [ 5862.662608] schedule_timeout+0xb4/0x190 [ 5862.663716] ? do_raw_spin_unlock+0x75/0x190 [ 5862.664989] ? __next_timer_interrupt+0x160/0x160 [ 5862.666352] ? prepare_to_swait_event+0xf3/0x140 [ 5862.669331] rcu_gp_kthread+0x8dd/0x15b0 [ 5862.670605] ? force_qs_rnp+0x2c0/0x2c0 [ 5862.672429] kthread+0x1d1/0x200 [ 5862.673444] ? set_kthread_struct+0x70/0x70 [ 5862.674588] ret_from_fork+0x1f/0x30 [ 5862.675553] rcu: Stack dump where RCU GP kthread last ran: [ 5862.677003] Sending NMI from CPU 1 to CPUs 3: [ 5862.679502] NMI backtrace for cpu 3 [ 5862.679508] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5862.679510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5862.679512] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5862.679513] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5862.679515] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 5862.679518] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5862.679519] RDX: 0000000000003915 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5862.679520] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5862.679522] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5862.679523] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5862.679524] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5862.679525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5862.679526] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5862.679527] Call Trace: [ 5862.679528] [ 5862.679529] ? show_regs.cold.9+0x22/0x2f [ 5862.679530] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5862.679531] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5862.679532] ? nmi_handle+0xbd/0x210 [ 5862.679533] ? default_do_nmi+0x4b/0x1b0 [ 5862.679534] ? do_nmi+0x280/0x380 [ 5862.679535] ? end_repeat_nmi+0x16/0x1f [ 5862.679536] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5862.679537] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5862.679538] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5862.679539] [ 5862.679539] do_raw_spin_lock+0x177/0x190 [ 5862.679540] _raw_spin_lock+0x21/0x30 [ 5862.679541] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5862.679542] obd_mod_free+0x46/0x140 [ptlrpc] [ 5862.679543] mdc_commit_open+0x91/0xb0 [mdc] [ 5862.679544] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5862.679545] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5862.679546] after_reply+0xe2c/0x1780 [ptlrpc] [ 5862.679547] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5862.679548] ? __next_timer_interrupt+0x160/0x160 [ 5862.679549] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5862.679550] ? do_wait_intr+0xf0/0xf0 [ 5862.679552] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5862.679553] kthread+0x1d1/0x200 [ 5862.679554] ? set_kthread_struct+0x70/0x70 [ 5862.679555] ret_from_fork+0x1f/0x30 [ 5864.053529] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 5864.058375] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 5864.062108] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5864.062191] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5864.062198] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5864.062201] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5864.067627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5864.067707] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5864.067714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5864.067717] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5864.067732] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5864.067737] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5864.067741] RAX: 000000000000760e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5864.067743] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5864.067746] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5864.067748] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5864.067750] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5864.067754] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5864.067757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5864.067759] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5864.067769] Call Trace: [ 5864.069526] [ 5864.108259] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5864.108281] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5864.108289] RAX: 00000000000066b2 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5864.108291] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5864.108294] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5864.108296] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5864.108298] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5864.108301] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5864.108304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5864.108306] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5864.108314] Call Trace: [ 5864.108320] [ 5864.108326] ? show_regs.cold.9+0x22/0x2f [ 5864.108340] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5864.108347] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5864.108351] ? watchdog+0x40/0x40 [ 5864.108354] ? hrtimer_interrupt+0x128/0x2c0 [ 5864.108358] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5864.108366] ? apic_timer_interrupt+0xf/0x20 [ 5864.108370] [ 5864.108373] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5864.108381] do_raw_spin_lock+0x177/0x190 [ 5864.108387] _raw_spin_lock+0x21/0x30 [ 5864.108391] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5864.108809] ? do_raw_spin_unlock+0x75/0x190 [ 5864.108815] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5864.109020] ? do_wait_intr+0xf0/0xf0 [ 5864.109028] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5864.109802] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5864.110020] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5864.110596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5864.110757] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5864.110802] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5864.110829] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5864.110856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5864.110936] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5864.113186] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5864.113912] lmv_intent_open+0x338/0x1150 [lmv] [ 5864.113944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5864.114076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5864.114516] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5864.114601] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5864.114705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5864.114789] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5864.136461] ? show_regs.cold.9+0x22/0x2f [ 5864.136484] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5864.136492] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5864.136497] ? watchdog+0x40/0x40 [ 5864.136502] ? hrtimer_interrupt+0x128/0x2c0 [ 5864.136507] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5864.136516] ? apic_timer_interrupt+0xf/0x20 [ 5864.136522] [ 5864.136527] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5864.136537] do_raw_spin_lock+0x177/0x190 [ 5864.136542] _raw_spin_lock+0x21/0x30 [ 5864.136547] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5864.145605] ? do_raw_spin_unlock+0x75/0x190 [ 5864.145628] ? _raw_spin_unlock+0x12/0x30 [ 5864.145635] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5864.145773] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5864.145795] ll_atomic_open+0x36a/0xd70 [lustre] [ 5864.145923] lookup_open+0x356/0xb40 [ 5864.145934] do_last+0x313/0xfc0 [ 5864.145939] ? nd_jump_root+0xe5/0x160 [ 5864.145944] ? path_init+0x437/0x520 [ 5864.145949] path_openat+0xf7/0x500 [ 5864.145954] ? do_raw_spin_unlock+0x75/0x190 [ 5864.145960] do_filp_open+0x99/0x140 [ 5864.145967] ? getname_flags+0x6e/0x330 [ 5864.145973] ? __check_object_size+0xff/0x256 [ 5864.145980] ? do_raw_spin_unlock+0x75/0x190 [ 5864.145984] ? _raw_spin_unlock+0x12/0x30 [ 5864.145989] do_sys_openat2+0x2b4/0x410 [ 5864.145993] do_sys_open+0x73/0xa0 [ 5864.145998] __x64_sys_openat+0x24/0x30 [ 5864.146002] do_syscall_64+0xc1/0x440 [ 5864.146009] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5864.146015] RIP: 0033:0x7f1a63983332 [ 5864.146023] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5864.146027] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5864.146032] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5864.146034] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5864.146037] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5864.146039] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5864.146412] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5864.713321] ? do_raw_spin_unlock+0x75/0x190 [ 5864.717801] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5864.725682] ? do_wait_intr+0xf0/0xf0 [ 5864.729187] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5864.732886] mdc_close+0x53b/0xd70 [mdc] [ 5864.735211] lmv_close+0x18d/0x490 [lmv] [ 5864.737395] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5864.742311] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5864.748114] ll_md_close+0x274/0x800 [lustre] [ 5864.750880] ll_file_release+0x128/0xc50 [lustre] [ 5864.759628] __fput+0x10d/0x440 [ 5864.760974] ____fput+0x12/0x20 [ 5864.762214] task_work_run+0xc6/0x110 [ 5864.775182] exit_to_usermode_loop+0x1dd/0x1f0 [ 5864.781031] do_syscall_64+0x3d6/0x440 [ 5864.794281] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5864.811267] RIP: 0033:0x7f1a63983cd8 [ 5864.816409] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5864.846546] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5864.867905] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5864.878338] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5864.900725] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5864.903457] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5864.922249] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5869.538360] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3846s! [ 5869.545653] Showing busy workqueues and worker pools: [ 5869.549698] workqueue events: flags=0x0 [ 5869.552677] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5869.558878] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5869.566795] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5869.571542] pending: sec_gc_main [ptlrpc] [ 5869.575527] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5869.581179] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5869.589077] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5869.600662] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5869.612300] workqueue events_long: flags=0x0 [ 5869.615419] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5869.623311] pending: lu_objects_flush [obdclass] [ 5869.629470] workqueue events_unbound: flags=0x2 [ 5869.632234] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5869.637342] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5869.646909] workqueue events_power_efficient: flags=0x80 [ 5869.656669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5869.668968] pending: neigh_periodic_work, do_cache_clean [ 5869.675319] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5869.679817] pending: neigh_periodic_work, check_lifetime [ 5869.686947] workqueue mm_percpu_wq: flags=0x8 [ 5869.699341] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5869.707572] pending: vmstat_update [ 5869.709024] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5869.723786] pending: vmstat_update [ 5869.725562] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5869.729068] pending: vmstat_update [ 5869.730612] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5869.738573] pending: vmstat_update [ 5869.741963] workqueue writeback: flags=0x4a [ 5869.747141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5869.752240] pending: wb_workfn, wb_workfn [ 5869.754790] workqueue xprtiod: flags=0xa [ 5869.757059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5869.762841] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5869.771065] workqueue ipv6_addrconf: flags=0x40008 [ 5869.773813] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5869.783725] pending: addrconf_verify_work [ 5869.791521] workqueue knbd0-recv: flags=0x1a [ 5869.798241] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5869.808620] in-flight: 647:recv_work [ 5869.813197] workqueue ptlrpc_pinger: flags=0x2 [ 5869.824260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5869.830915] pending: ptlrpc_pinger_main [ptlrpc] [ 5869.836281] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5859s workers=2 idle: 652 [ 5888.056008] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5888.058008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5888.063229] Modules linked in: [ 5888.066368] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [ 5888.071669] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [ 5888.074851] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 5888.084510] ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5888.084607] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5888.084623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5888.084626] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5888.084709] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5888.084714] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5888.084718] RAX: 000000000000290a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5888.084720] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5888.084723] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5888.084725] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5888.084727] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5888.084734] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5888.084737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5888.084740] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5888.084748] Call Trace: [ 5888.084765] [ 5888.084771] ? show_regs.cold.9+0x22/0x2f [ 5888.084779] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5888.084786] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5888.084794] ? watchdog+0x40/0x40 [ 5888.084798] ? hrtimer_interrupt+0x128/0x2c0 [ 5888.084803] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5888.084828] ? apic_timer_interrupt+0xf/0x20 [ 5888.084833] [ 5888.084835] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5888.084841] do_raw_spin_lock+0x177/0x190 [ 5888.084845] _raw_spin_lock+0x21/0x30 [ 5888.084849] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5888.085165] ? do_raw_spin_unlock+0x75/0x190 [ 5888.085171] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5888.085372] ? do_wait_intr+0xf0/0xf0 [ 5888.085379] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5888.085569] mdc_close+0x53b/0xd70 [mdc] [ 5888.085608] lmv_close+0x18d/0x490 [lmv] [ 5888.085642] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5888.085923] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5888.086007] ll_md_close+0x274/0x800 [lustre] [ 5888.086089] ll_file_release+0x128/0xc50 [lustre] [ 5888.086143] __fput+0x10d/0x440 [ 5888.086150] ____fput+0x12/0x20 [ 5888.086153] task_work_run+0xc6/0x110 [ 5888.086159] exit_to_usermode_loop+0x1dd/0x1f0 [ 5888.086164] do_syscall_64+0x3d6/0x440 [ 5888.086168] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5888.086173] RIP: 0033:0x7f1a63983cd8 [ 5888.086188] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5888.086191] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5888.086194] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5888.086196] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5888.086197] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5888.086199] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5888.086200] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5888.299309] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5888.317386] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5888.324541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5888.331090] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5888.333599] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5888.341233] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5888.343500] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5888.345641] RDX: 00000000000026bc RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5888.347998] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5888.350395] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5888.352951] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5888.355636] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5888.358576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5888.360585] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5888.364738] Call Trace: [ 5888.371627] [ 5888.375006] ? show_regs.cold.9+0x22/0x2f [ 5888.377551] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5888.380969] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5888.385102] ? watchdog+0x40/0x40 [ 5888.388330] ? hrtimer_interrupt+0x128/0x2c0 [ 5888.390077] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5888.394385] ? apic_timer_interrupt+0xf/0x20 [ 5888.399290] [ 5888.400856] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5888.402438] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5888.407416] do_raw_spin_lock+0x177/0x190 [ 5888.409467] _raw_spin_lock+0x21/0x30 [ 5888.416098] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5888.420327] obd_mod_free+0x46/0x140 [ptlrpc] [ 5888.423478] mdc_commit_open+0x91/0xb0 [mdc] [ 5888.425423] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5888.429335] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5888.436122] after_reply+0xe2c/0x1780 [ptlrpc] [ 5888.441434] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5888.445490] ? __next_timer_interrupt+0x160/0x160 [ 5888.447347] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5888.451232] ? do_wait_intr+0xf0/0xf0 [ 5888.454603] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5888.463611] kthread+0x1d1/0x200 [ 5888.468654] ? set_kthread_struct+0x70/0x70 [ 5888.472632] ret_from_fork+0x1f/0x30 [ 5892.059885] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5892.067527] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5892.086694] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5892.090488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5892.093444] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5892.095586] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5892.102918] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5892.105827] RAX: 0000000000000bb1 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5892.108685] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5892.111596] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5892.114478] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5892.117318] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5892.120747] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5892.124115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5892.126648] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5892.129241] Call Trace: [ 5892.130649] [ 5892.131400] ? show_regs.cold.9+0x22/0x2f [ 5892.132903] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5892.134897] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5892.136893] ? watchdog+0x40/0x40 [ 5892.138349] ? hrtimer_interrupt+0x128/0x2c0 [ 5892.139987] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5892.142003] ? apic_timer_interrupt+0xf/0x20 [ 5892.143791] [ 5892.144560] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5892.146766] do_raw_spin_lock+0x177/0x190 [ 5892.148462] _raw_spin_lock+0x21/0x30 [ 5892.149845] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5892.152156] ? do_raw_spin_unlock+0x75/0x190 [ 5892.153512] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5892.155505] ? do_wait_intr+0xf0/0xf0 [ 5892.156869] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5892.159549] mdc_close+0x53b/0xd70 [mdc] [ 5892.160959] lmv_close+0x18d/0x490 [lmv] [ 5892.162552] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5892.164757] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5892.166772] ll_md_close+0x274/0x800 [lustre] [ 5892.168685] ll_file_release+0x128/0xc50 [lustre] [ 5892.170437] __fput+0x10d/0x440 [ 5892.171888] ____fput+0x12/0x20 [ 5892.173164] task_work_run+0xc6/0x110 [ 5892.174627] exit_to_usermode_loop+0x1dd/0x1f0 [ 5892.176182] do_syscall_64+0x3d6/0x440 [ 5892.177743] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5892.179878] RIP: 0033:0x7f1a63983cd8 [ 5892.180996] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5892.190968] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5892.194926] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5892.200548] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5892.203528] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5892.208976] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5892.214103] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5896.053106] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 5896.055860] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5896.121988] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5896.136369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5896.148085] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5896.158976] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5896.183695] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5896.189343] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5896.198497] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5896.204805] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5896.211472] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5896.214707] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5896.219197] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5896.227275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5896.230331] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5896.234358] Call Trace: [ 5896.235697] [ 5896.236920] ? show_regs.cold.9+0x22/0x2f [ 5896.239097] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5896.247062] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5896.250134] ? watchdog+0x40/0x40 [ 5896.255897] ? hrtimer_interrupt+0x128/0x2c0 [ 5896.258130] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5896.263458] ? apic_timer_interrupt+0xf/0x20 [ 5896.264947] [ 5896.265659] ? native_safe_halt+0x2e/0x50 [ 5896.268496] kvm_wait+0xcd/0xe0 [ 5896.273059] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5896.281549] do_raw_spin_lock+0x177/0x190 [ 5896.285836] _raw_spin_lock+0x21/0x30 [ 5896.290818] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5896.296421] ? do_raw_spin_unlock+0x75/0x190 [ 5896.300675] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5896.304537] ? do_wait_intr+0xf0/0xf0 [ 5896.305686] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5896.319591] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5896.327397] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5896.334219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5896.345442] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5896.355914] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5896.366665] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5896.368169] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5896.376233] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5896.382895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5896.398957] lmv_intent_open+0x338/0x1150 [lmv] [ 5896.400732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5896.413964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5896.427101] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5896.428599] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5896.444421] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5896.446080] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5896.457659] ? do_raw_spin_unlock+0x75/0x190 [ 5896.464872] ? _raw_spin_unlock+0x12/0x30 [ 5896.474382] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5896.486294] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5896.496533] ll_atomic_open+0x36a/0xd70 [lustre] [ 5896.506352] lookup_open+0x356/0xb40 [ 5896.514531] do_last+0x313/0xfc0 [ 5896.520414] ? nd_jump_root+0xe5/0x160 [ 5896.528379] ? path_init+0x437/0x520 [ 5896.540683] path_openat+0xf7/0x500 [ 5896.542007] ? do_raw_spin_unlock+0x75/0x190 [ 5896.543430] do_filp_open+0x99/0x140 [ 5896.545340] ? getname_flags+0x6e/0x330 [ 5896.547310] ? __check_object_size+0xff/0x256 [ 5896.549151] ? do_raw_spin_unlock+0x75/0x190 [ 5896.550852] ? _raw_spin_unlock+0x12/0x30 [ 5896.552263] do_sys_openat2+0x2b4/0x410 [ 5896.554177] do_sys_open+0x73/0xa0 [ 5896.560316] __x64_sys_openat+0x24/0x30 [ 5896.561934] do_syscall_64+0xc1/0x440 [ 5896.563150] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5896.570789] RIP: 0033:0x7f1a63983332 [ 5896.574151] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5896.590887] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5896.593261] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5896.598216] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5896.606196] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5896.615431] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5896.630718] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5900.257710] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3877s! [ 5900.265285] Showing busy workqueues and worker pools: [ 5900.267945] workqueue events: flags=0x0 [ 5900.270041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5900.278077] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5900.281838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5900.284889] pending: sec_gc_main [ptlrpc] [ 5900.287255] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5900.294503] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5900.299822] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5900.303346] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5900.307614] workqueue events_long: flags=0x0 [ 5900.309941] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5900.312921] pending: lu_objects_flush [obdclass] [ 5900.315263] workqueue events_unbound: flags=0x2 [ 5900.317527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5900.321561] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5900.332475] workqueue events_power_efficient: flags=0x80 [ 5900.337649] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5900.343301] pending: neigh_periodic_work, do_cache_clean [ 5900.347531] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5900.354158] pending: neigh_periodic_work, check_lifetime [ 5900.362506] workqueue mm_percpu_wq: flags=0x8 [ 5900.365638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5900.372378] pending: vmstat_update [ 5900.375931] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5900.379044] pending: vmstat_update [ 5900.380430] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5900.389272] pending: vmstat_update [ 5900.392579] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5900.397584] pending: vmstat_update [ 5900.401610] workqueue writeback: flags=0x4a [ 5900.405863] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5900.409941] pending: wb_workfn, wb_workfn [ 5900.413275] workqueue xprtiod: flags=0xa [ 5900.415660] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5900.421657] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5900.427200] workqueue ipv6_addrconf: flags=0x40008 [ 5900.431447] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5900.436022] pending: addrconf_verify_work [ 5900.437993] workqueue knbd0-recv: flags=0x1a [ 5900.441577] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5900.444597] in-flight: 647:recv_work [ 5900.446600] workqueue ptlrpc_pinger: flags=0x2 [ 5900.448890] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5900.452152] pending: ptlrpc_pinger_main [ptlrpc] [ 5900.455098] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5890s workers=2 idle: 652 [ 5916.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5916.059405] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5916.065251] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5916.065337] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5916.065344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5916.065347] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5916.065360] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5916.065365] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5916.065369] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5916.065372] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5916.065374] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5916.065377] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5916.065380] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5916.065383] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5916.065386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5916.065388] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5916.065404] Call Trace: [ 5916.065409] [ 5916.065415] ? show_regs.cold.9+0x22/0x2f [ 5916.065424] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5916.065430] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5916.065434] ? watchdog+0x40/0x40 [ 5916.065438] ? hrtimer_interrupt+0x128/0x2c0 [ 5916.065442] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5916.065450] ? apic_timer_interrupt+0xf/0x20 [ 5916.065455] [ 5916.065458] ? native_safe_halt+0x2e/0x50 [ 5916.065461] kvm_wait+0xcd/0xe0 [ 5916.065469] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5916.065475] do_raw_spin_lock+0x177/0x190 [ 5916.065479] _raw_spin_lock+0x21/0x30 [ 5916.065484] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5916.065781] obd_mod_free+0x46/0x140 [ptlrpc] [ 5916.068260] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5916.068313] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5916.068318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5916.068320] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5916.068329] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5916.068332] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5916.068335] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5916.068337] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5916.068338] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5916.068340] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5916.068341] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5916.068344] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5916.068346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5916.068347] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5916.068353] Call Trace: [ 5916.068356] [ 5916.068360] ? show_regs.cold.9+0x22/0x2f [ 5916.068367] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5916.068371] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5916.068374] ? watchdog+0x40/0x40 [ 5916.068377] ? hrtimer_interrupt+0x128/0x2c0 [ 5916.068380] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5916.068385] ? apic_timer_interrupt+0xf/0x20 [ 5916.068388] [ 5916.068390] ? native_safe_halt+0x2e/0x50 [ 5916.068392] kvm_wait+0xcd/0xe0 [ 5916.068398] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5916.068403] do_raw_spin_lock+0x177/0x190 [ 5916.068406] _raw_spin_lock+0x21/0x30 [ 5916.068409] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5916.068653] ? do_raw_spin_unlock+0x75/0x190 [ 5916.071981] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5916.094212] mdc_commit_open+0x91/0xb0 [mdc] [ 5916.099727] ? do_wait_intr+0xf0/0xf0 [ 5916.099750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5916.106759] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5916.113844] mdc_close+0x53b/0xd70 [mdc] [ 5916.113893] lmv_close+0x18d/0x490 [lmv] [ 5916.124268] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5916.125847] after_reply+0xe2c/0x1780 [ptlrpc] [ 5916.126831] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5916.132041] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5916.132167] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5916.132251] ll_md_close+0x274/0x800 [lustre] [ 5916.132336] ll_file_release+0x128/0xc50 [lustre] [ 5916.132419] __fput+0x10d/0x440 [ 5916.132429] ____fput+0x12/0x20 [ 5916.132434] task_work_run+0xc6/0x110 [ 5916.132442] exit_to_usermode_loop+0x1dd/0x1f0 [ 5916.132449] do_syscall_64+0x3d6/0x440 [ 5916.132454] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5916.132462] RIP: 0033:0x7f1a63983cd8 [ 5916.132470] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5916.132474] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5916.132479] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5916.132481] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5916.132483] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5916.132485] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5916.132487] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5916.453080] ? __next_timer_interrupt+0x160/0x160 [ 5916.455059] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5916.457135] ? do_wait_intr+0xf0/0xf0 [ 5916.459925] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5916.463197] kthread+0x1d1/0x200 [ 5916.464683] ? set_kthread_struct+0x70/0x70 [ 5916.465850] ret_from_fork+0x1f/0x30 [ 5920.064161] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5920.078183] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5920.158461] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5920.172218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5920.183475] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5920.187140] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5920.213454] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5920.221291] RAX: 0000000000007e0d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5920.231810] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5920.242760] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5920.250963] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5920.268220] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5920.275027] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5920.286968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5920.290108] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5920.294109] Call Trace: [ 5920.295786] [ 5920.296597] ? show_regs.cold.9+0x22/0x2f [ 5920.299186] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5920.301437] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5920.303196] ? watchdog+0x40/0x40 [ 5920.304949] ? hrtimer_interrupt+0x128/0x2c0 [ 5920.309373] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5920.315902] ? apic_timer_interrupt+0xf/0x20 [ 5920.317441] [ 5920.318159] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5920.325404] do_raw_spin_lock+0x177/0x190 [ 5920.327887] _raw_spin_lock+0x21/0x30 [ 5920.330241] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5920.335571] ? do_raw_spin_unlock+0x75/0x190 [ 5920.337926] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5920.343734] ? do_wait_intr+0xf0/0xf0 [ 5920.347609] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5920.351600] mdc_close+0x53b/0xd70 [mdc] [ 5920.355764] lmv_close+0x18d/0x490 [lmv] [ 5920.358424] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5920.362526] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5920.365121] ll_md_close+0x274/0x800 [lustre] [ 5920.374015] ll_file_release+0x128/0xc50 [lustre] [ 5920.376713] __fput+0x10d/0x440 [ 5920.379712] ____fput+0x12/0x20 [ 5920.384243] task_work_run+0xc6/0x110 [ 5920.385671] exit_to_usermode_loop+0x1dd/0x1f0 [ 5920.391839] do_syscall_64+0x3d6/0x440 [ 5920.395967] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5920.398658] RIP: 0033:0x7f1a63983cd8 [ 5920.403883] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5920.418416] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5920.421918] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5920.428710] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5920.431830] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5920.438351] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5920.450793] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5928.055309] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5928.060670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5928.083358] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5928.088009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5928.092285] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5928.094826] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5928.103919] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5928.108055] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5928.110953] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5928.113202] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5928.116115] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5928.120124] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5928.123963] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5928.127996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5928.130356] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5928.133289] Call Trace: [ 5928.134369] [ 5928.135284] ? show_regs.cold.9+0x22/0x2f [ 5928.137088] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5928.139155] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5928.141249] ? watchdog+0x40/0x40 [ 5928.142826] ? hrtimer_interrupt+0x128/0x2c0 [ 5928.144742] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5928.146928] ? apic_timer_interrupt+0xf/0x20 [ 5928.148915] [ 5928.149836] ? native_safe_halt+0x2e/0x50 [ 5928.151523] kvm_wait+0xcd/0xe0 [ 5928.152910] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5928.156782] do_raw_spin_lock+0x177/0x190 [ 5928.158921] _raw_spin_lock+0x21/0x30 [ 5928.161073] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5928.169213] ? do_raw_spin_unlock+0x75/0x190 [ 5928.174014] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5928.178904] ? do_wait_intr+0xf0/0xf0 [ 5928.180929] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5928.185632] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5928.189148] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5928.191963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5928.194606] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5928.197340] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5928.199999] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5928.204026] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5928.209842] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5928.213401] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5928.217208] lmv_intent_open+0x338/0x1150 [lmv] [ 5928.219911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5928.222555] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5928.225523] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5928.227222] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5928.229770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5928.232537] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5928.234368] ? do_raw_spin_unlock+0x75/0x190 [ 5928.237286] ? _raw_spin_unlock+0x12/0x30 [ 5928.238860] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5928.241210] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5928.247583] ll_atomic_open+0x36a/0xd70 [lustre] [ 5928.249825] lookup_open+0x356/0xb40 [ 5928.252746] do_last+0x313/0xfc0 [ 5928.255988] ? nd_jump_root+0xe5/0x160 [ 5928.257766] ? path_init+0x437/0x520 [ 5928.259587] path_openat+0xf7/0x500 [ 5928.261869] ? do_raw_spin_unlock+0x75/0x190 [ 5928.264807] do_filp_open+0x99/0x140 [ 5928.266208] ? getname_flags+0x6e/0x330 [ 5928.268025] ? __check_object_size+0xff/0x256 [ 5928.270291] ? do_raw_spin_unlock+0x75/0x190 [ 5928.272504] ? _raw_spin_unlock+0x12/0x30 [ 5928.274147] do_sys_openat2+0x2b4/0x410 [ 5928.275755] do_sys_open+0x73/0xa0 [ 5928.276973] __x64_sys_openat+0x24/0x30 [ 5928.278484] do_syscall_64+0xc1/0x440 [ 5928.279953] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5928.281888] RIP: 0033:0x7f1a63983332 [ 5928.283658] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5928.290627] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5928.292967] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5928.295630] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5928.298522] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5928.301464] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5928.304979] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5930.465166] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3907s! [ 5930.471556] Showing busy workqueues and worker pools: [ 5930.473613] workqueue events: flags=0x0 [ 5930.475217] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5930.477940] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5930.482659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5930.485508] pending: sec_gc_main [ptlrpc] [ 5930.487559] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5930.490095] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5930.497535] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5930.503719] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5930.507200] workqueue events_long: flags=0x0 [ 5930.508738] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5930.511025] pending: lu_objects_flush [obdclass] [ 5930.513053] workqueue events_unbound: flags=0x2 [ 5930.515021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5930.517105] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5930.520821] workqueue events_power_efficient: flags=0x80 [ 5930.522809] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5930.525472] pending: neigh_periodic_work, do_cache_clean [ 5930.527514] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5930.530076] pending: neigh_periodic_work, check_lifetime [ 5930.531887] workqueue mm_percpu_wq: flags=0x8 [ 5930.533725] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5930.536589] pending: vmstat_update [ 5930.538606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5930.541136] pending: vmstat_update [ 5930.542699] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5930.545260] pending: vmstat_update [ 5930.546707] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5930.549276] pending: vmstat_update [ 5930.551013] workqueue writeback: flags=0x4a [ 5930.552770] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5930.555453] pending: wb_workfn, wb_workfn [ 5930.557313] workqueue xprtiod: flags=0xa [ 5930.558831] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5930.561230] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5930.564836] workqueue ipv6_addrconf: flags=0x40008 [ 5930.567257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5930.570125] pending: addrconf_verify_work [ 5930.571732] workqueue knbd0-recv: flags=0x1a [ 5930.573378] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5930.576172] in-flight: 647:recv_work [ 5930.577807] workqueue ptlrpc_pinger: flags=0x2 [ 5930.579861] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5930.582176] pending: ptlrpc_pinger_main [ptlrpc] [ 5930.584807] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5920s workers=2 idle: 652 [ 5944.058017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5944.058111] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5944.064060] Modules linked in: lustre(O) [ 5944.072343] Modules linked in: [ 5944.072345] osp(O) ofd(O) lod(O) [ 5944.075372] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5944.075449] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5944.075456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5944.075458] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5944.075473] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5944.075478] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5944.075482] RAX: 0000000000002b75 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5944.075484] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5944.075487] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5944.075489] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5944.075491] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5944.075673] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5944.075677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5944.075679] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5944.075688] Call Trace: [ 5944.075693] [ 5944.075698] ? show_regs.cold.9+0x22/0x2f [ 5944.075707] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5944.075713] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5944.075717] ? watchdog+0x40/0x40 [ 5944.075720] ? hrtimer_interrupt+0x128/0x2c0 [ 5944.075724] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5944.075732] ? apic_timer_interrupt+0xf/0x20 [ 5944.075737] [ 5944.075740] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5944.075745] do_raw_spin_lock+0x177/0x190 [ 5944.075750] _raw_spin_lock+0x21/0x30 [ 5944.075754] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5944.077360] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5944.077419] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5944.077426] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5944.079447] ? do_raw_spin_unlock+0x75/0x190 [ 5944.079464] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5944.080260] ? do_wait_intr+0xf0/0xf0 [ 5944.080269] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5944.080482] mdc_close+0x53b/0xd70 [mdc] [ 5944.080691] lmv_close+0x18d/0x490 [lmv] [ 5944.080724] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5944.080835] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5944.081049] ll_md_close+0x274/0x800 [lustre] [ 5944.081138] ll_file_release+0x128/0xc50 [lustre] [ 5944.081229] __fput+0x10d/0x440 [ 5944.081238] ____fput+0x12/0x20 [ 5944.081244] task_work_run+0xc6/0x110 [ 5944.081252] exit_to_usermode_loop+0x1dd/0x1f0 [ 5944.081259] do_syscall_64+0x3d6/0x440 [ 5944.081264] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5944.081272] RIP: 0033:0x7f1a63983cd8 [ 5944.081290] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5944.081293] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5944.081297] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5944.081300] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5944.081302] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5944.081305] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5944.081307] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5944.309596] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5944.312137] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5944.319658] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5944.322757] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5944.325634] RDX: 0000000000003bc9 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5944.328579] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5944.331386] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5944.334385] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5944.337215] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5944.340724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5944.343067] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5944.346049] Call Trace: [ 5944.347117] [ 5944.348074] ? show_regs.cold.9+0x22/0x2f [ 5944.349895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5944.351994] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5944.353976] ? watchdog+0x40/0x40 [ 5944.355239] ? hrtimer_interrupt+0x128/0x2c0 [ 5944.357146] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5944.358922] ? apic_timer_interrupt+0xf/0x20 [ 5944.360447] [ 5944.361368] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5944.363934] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5944.366827] do_raw_spin_lock+0x177/0x190 [ 5944.370404] _raw_spin_lock+0x21/0x30 [ 5944.371651] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5944.379343] obd_mod_free+0x46/0x140 [ptlrpc] [ 5944.384351] mdc_commit_open+0x91/0xb0 [mdc] [ 5944.387130] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5944.392797] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5944.398114] after_reply+0xe2c/0x1780 [ptlrpc] [ 5944.403615] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5944.408076] ? __next_timer_interrupt+0x160/0x160 [ 5944.410572] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5944.412452] ? do_wait_intr+0xf0/0xf0 [ 5944.414099] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5944.416989] kthread+0x1d1/0x200 [ 5944.418736] ? set_kthread_struct+0x70/0x70 [ 5944.420753] ret_from_fork+0x1f/0x30 [ 5948.057044] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5948.059714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5948.105154] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5948.114405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5948.118038] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5948.121001] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5948.131880] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5948.137820] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5948.140189] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5948.147092] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5948.150978] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5948.157044] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5948.162562] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5948.169567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5948.172496] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5948.176022] Call Trace: [ 5948.177409] [ 5948.178787] ? show_regs.cold.9+0x22/0x2f [ 5948.183419] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5948.185218] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5948.189607] ? watchdog+0x40/0x40 [ 5948.190933] ? hrtimer_interrupt+0x128/0x2c0 [ 5948.193691] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5948.196702] ? apic_timer_interrupt+0xf/0x20 [ 5948.203925] [ 5948.205444] ? native_safe_halt+0x2e/0x50 [ 5948.207368] kvm_wait+0xcd/0xe0 [ 5948.209154] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5948.211667] do_raw_spin_lock+0x177/0x190 [ 5948.213661] _raw_spin_lock+0x21/0x30 [ 5948.220844] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5948.224899] ? do_raw_spin_unlock+0x75/0x190 [ 5948.227886] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5948.231042] ? do_wait_intr+0xf0/0xf0 [ 5948.238940] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5948.241094] mdc_close+0x53b/0xd70 [mdc] [ 5948.249224] lmv_close+0x18d/0x490 [lmv] [ 5948.254571] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5948.259583] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5948.267370] ll_md_close+0x274/0x800 [lustre] [ 5948.277526] ll_file_release+0x128/0xc50 [lustre] [ 5948.285033] __fput+0x10d/0x440 [ 5948.286624] ____fput+0x12/0x20 [ 5948.288176] task_work_run+0xc6/0x110 [ 5948.290011] exit_to_usermode_loop+0x1dd/0x1f0 [ 5948.295624] do_syscall_64+0x3d6/0x440 [ 5948.297497] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5948.299715] RIP: 0033:0x7f1a63983cd8 [ 5948.301144] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5948.315183] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5948.321627] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5948.329522] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5948.337612] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5948.344742] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5948.347760] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5956.053837] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 5956.060419] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5956.132223] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5956.148671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5956.158799] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5956.161095] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5956.192235] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5956.200783] RAX: 0000000000004285 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5956.208384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5956.212789] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5956.218613] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5956.224613] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5956.232612] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5956.241067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5956.255379] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5956.268969] Call Trace: [ 5956.270792] [ 5956.271979] ? show_regs.cold.9+0x22/0x2f [ 5956.278735] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5956.285830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5956.290673] ? watchdog+0x40/0x40 [ 5956.291912] ? hrtimer_interrupt+0x128/0x2c0 [ 5956.295147] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5956.299615] ? apic_timer_interrupt+0xf/0x20 [ 5956.301256] [ 5956.302044] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5956.307246] do_raw_spin_lock+0x177/0x190 [ 5956.312668] _raw_spin_lock+0x21/0x30 [ 5956.315850] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5956.325824] ? do_raw_spin_unlock+0x75/0x190 [ 5956.327440] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5956.351398] ? do_wait_intr+0xf0/0xf0 [ 5956.359868] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5956.370950] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5956.376431] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5956.390676] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5956.407985] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5956.410110] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5956.422589] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5956.425512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5956.427764] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5956.432473] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5956.434212] lmv_intent_open+0x338/0x1150 [lmv] [ 5956.436533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5956.438441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5956.441364] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5956.443596] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5956.445631] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5956.448019] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5956.450436] ? do_raw_spin_unlock+0x75/0x190 [ 5956.452435] ? _raw_spin_unlock+0x12/0x30 [ 5956.454332] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5956.456633] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5956.460236] ll_atomic_open+0x36a/0xd70 [lustre] [ 5956.466286] lookup_open+0x356/0xb40 [ 5956.467961] do_last+0x313/0xfc0 [ 5956.469378] ? nd_jump_root+0xe5/0x160 [ 5956.471269] ? path_init+0x437/0x520 [ 5956.475562] path_openat+0xf7/0x500 [ 5956.477543] ? do_raw_spin_unlock+0x75/0x190 [ 5956.480942] do_filp_open+0x99/0x140 [ 5956.482967] ? getname_flags+0x6e/0x330 [ 5956.484763] ? __check_object_size+0xff/0x256 [ 5956.486626] ? do_raw_spin_unlock+0x75/0x190 [ 5956.488627] ? _raw_spin_unlock+0x12/0x30 [ 5956.490771] do_sys_openat2+0x2b4/0x410 [ 5956.492648] do_sys_open+0x73/0xa0 [ 5956.497140] __x64_sys_openat+0x24/0x30 [ 5956.502944] do_syscall_64+0xc1/0x440 [ 5956.506227] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5956.508709] RIP: 0033:0x7f1a63983332 [ 5956.510437] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5956.525070] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5956.530645] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5956.535840] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5956.544510] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5956.551422] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5956.554975] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5960.673038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3937s! [ 5960.677822] Showing busy workqueues and worker pools: [ 5960.680804] workqueue events: flags=0x0 [ 5960.694450] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5960.707096] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5960.727741] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5960.733572] pending: sec_gc_main [ptlrpc] [ 5960.736903] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5960.749571] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5960.760678] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5960.770383] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5960.777197] workqueue events_long: flags=0x0 [ 5960.779808] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5960.792534] pending: lu_objects_flush [obdclass] [ 5960.796554] workqueue events_unbound: flags=0x2 [ 5960.802490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5960.814841] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5960.820522] workqueue events_power_efficient: flags=0x80 [ 5960.827791] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5960.835196] pending: neigh_periodic_work, do_cache_clean [ 5960.839833] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5960.850187] pending: neigh_periodic_work, check_lifetime [ 5960.853561] workqueue mm_percpu_wq: flags=0x8 [ 5960.856164] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5960.862008] pending: vmstat_update [ 5960.864936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5960.871950] pending: vmstat_update [ 5960.876809] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5960.881419] pending: vmstat_update [ 5960.883842] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5960.888923] pending: vmstat_update [ 5960.894836] workqueue writeback: flags=0x4a [ 5960.899945] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5960.907934] pending: wb_workfn, wb_workfn [ 5960.912235] workqueue xprtiod: flags=0xa [ 5960.915444] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5960.925944] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5960.928901] workqueue ipv6_addrconf: flags=0x40008 [ 5960.940446] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5960.955764] pending: addrconf_verify_work [ 5960.963092] workqueue knbd0-recv: flags=0x1a [ 5960.967871] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5960.975772] in-flight: 647:recv_work [ 5960.979487] workqueue ptlrpc_pinger: flags=0x2 [ 5960.982305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5960.987795] pending: ptlrpc_pinger_main [ptlrpc] [ 5960.995631] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5951s workers=2 idle: 652 [ 5972.057420] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 5972.062976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5972.068643] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 5972.093248] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5972.093262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5972.093265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5972.093280] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5972.093284] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5972.093289] RAX: 0000000000007884 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5972.093292] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 5972.093294] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5972.093296] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 5972.093299] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 5972.093302] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 5972.093305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5972.093307] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 5972.093317] Call Trace: [ 5972.093322] [ 5972.093327] ? show_regs.cold.9+0x22/0x2f [ 5972.093336] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5972.093343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5972.093347] ? watchdog+0x40/0x40 [ 5972.093351] ? hrtimer_interrupt+0x128/0x2c0 [ 5972.093355] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5972.093365] ? apic_timer_interrupt+0xf/0x20 [ 5972.093370] [ 5972.093373] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5972.093379] do_raw_spin_lock+0x177/0x190 [ 5972.093384] _raw_spin_lock+0x21/0x30 [ 5972.093389] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5972.093702] ? do_raw_spin_unlock+0x75/0x190 [ 5972.093708] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5972.093966] ? do_wait_intr+0xf0/0xf0 [ 5972.093973] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5972.096640] mdc_close+0x53b/0xd70 [mdc] [ 5972.096695] lmv_close+0x18d/0x490 [lmv] [ 5972.096727] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5972.096852] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5972.096932] ll_md_close+0x274/0x800 [lustre] [ 5972.098099] ll_file_release+0x128/0xc50 [lustre] [ 5972.098199] __fput+0x10d/0x440 [ 5972.108382] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5972.108458] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5972.108465] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5972.108468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5972.108482] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5972.108486] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5972.108491] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 5972.108493] RDX: 0000000000006850 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 5972.108496] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 5972.108498] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 5972.108500] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5972.108504] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 5972.108506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5972.108509] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 5972.108518] Call Trace: [ 5972.108523] [ 5972.108529] ? show_regs.cold.9+0x22/0x2f [ 5972.108539] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5972.108545] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5972.108550] ? watchdog+0x40/0x40 [ 5972.108554] ? hrtimer_interrupt+0x128/0x2c0 [ 5972.108558] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5972.108566] ? apic_timer_interrupt+0xf/0x20 [ 5972.108570] [ 5972.108573] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5972.108579] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5972.108584] do_raw_spin_lock+0x177/0x190 [ 5972.108588] _raw_spin_lock+0x21/0x30 [ 5972.108593] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5972.120665] ____fput+0x12/0x20 [ 5972.121273] task_work_run+0xc6/0x110 [ 5972.121281] exit_to_usermode_loop+0x1dd/0x1f0 [ 5972.121288] do_syscall_64+0x3d6/0x440 [ 5972.121292] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5972.121300] RIP: 0033:0x7f1a63983cd8 [ 5972.121308] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5972.121312] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5972.121316] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 5972.121319] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5972.121321] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 5972.121323] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 5972.121325] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5972.803777] obd_mod_free+0x46/0x140 [ptlrpc] [ 5972.812251] mdc_commit_open+0x91/0xb0 [mdc] [ 5972.820813] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5972.826830] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5972.837231] after_reply+0xe2c/0x1780 [ptlrpc] [ 5972.846881] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5972.853823] ? __next_timer_interrupt+0x160/0x160 [ 5972.859573] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5972.864764] ? do_wait_intr+0xf0/0xf0 [ 5972.866625] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5972.871580] kthread+0x1d1/0x200 [ 5972.874324] ? set_kthread_struct+0x70/0x70 [ 5972.875358] ret_from_fork+0x1f/0x30 [ 5976.057116] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 5976.061339] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5976.166936] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5976.179777] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5976.201491] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5976.215355] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5976.258702] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5976.269627] RAX: 0000000000003813 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5976.281191] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 5976.295581] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 5976.303723] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 5976.312113] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 5976.317437] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 5976.326312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5976.331890] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 5976.337897] Call Trace: [ 5976.339950] [ 5976.341532] ? show_regs.cold.9+0x22/0x2f [ 5976.358615] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5976.374380] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5976.376154] ? watchdog+0x40/0x40 [ 5976.385399] ? hrtimer_interrupt+0x128/0x2c0 [ 5976.389943] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5976.398310] ? apic_timer_interrupt+0xf/0x20 [ 5976.399899] [ 5976.406552] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5976.411252] do_raw_spin_lock+0x177/0x190 [ 5976.415291] _raw_spin_lock+0x21/0x30 [ 5976.417697] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5976.428311] ? do_raw_spin_unlock+0x75/0x190 [ 5976.435371] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5976.437592] ? do_wait_intr+0xf0/0xf0 [ 5976.447492] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5976.458916] mdc_close+0x53b/0xd70 [mdc] [ 5976.466497] lmv_close+0x18d/0x490 [lmv] [ 5976.473149] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5976.479754] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5976.483161] ll_md_close+0x274/0x800 [lustre] [ 5976.497539] ll_file_release+0x128/0xc50 [lustre] [ 5976.508488] __fput+0x10d/0x440 [ 5976.515344] ____fput+0x12/0x20 [ 5976.522156] task_work_run+0xc6/0x110 [ 5976.529309] exit_to_usermode_loop+0x1dd/0x1f0 [ 5976.539437] do_syscall_64+0x3d6/0x440 [ 5976.540557] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5976.548853] RIP: 0033:0x7f1a63983cd8 [ 5976.554526] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5976.598662] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5976.632135] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 5976.638911] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 5976.655528] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 5976.668527] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 5976.673287] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5988.053214] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 5988.056633] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5988.090799] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5988.098450] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5988.107723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5988.111356] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5988.126004] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5988.136818] RAX: 0000000000003a12 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 5988.143177] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 5988.146628] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 5988.150452] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 5988.153719] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 5988.164855] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 5988.167898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5988.176243] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 5988.180117] Call Trace: [ 5988.182033] [ 5988.182768] ? show_regs.cold.9+0x22/0x2f [ 5988.184624] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5988.190829] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5988.202271] ? watchdog+0x40/0x40 [ 5988.208329] ? hrtimer_interrupt+0x128/0x2c0 [ 5988.211411] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5988.214178] ? apic_timer_interrupt+0xf/0x20 [ 5988.219009] [ 5988.220078] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5988.224561] do_raw_spin_lock+0x177/0x190 [ 5988.227935] _raw_spin_lock+0x21/0x30 [ 5988.229993] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5988.233551] ? do_raw_spin_unlock+0x75/0x190 [ 5988.235784] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5988.238909] ? do_wait_intr+0xf0/0xf0 [ 5988.240913] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5988.247187] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5988.250732] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5988.253965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.256665] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5988.259937] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5988.262413] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5988.264547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.269009] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5988.277496] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5988.280420] lmv_intent_open+0x338/0x1150 [lmv] [ 5988.286632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.297628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.303887] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5988.308665] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5988.310219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.316473] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5988.321609] ? do_raw_spin_unlock+0x75/0x190 [ 5988.323670] ? _raw_spin_unlock+0x12/0x30 [ 5988.326224] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5988.333155] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5988.340763] ll_atomic_open+0x36a/0xd70 [lustre] [ 5988.344171] lookup_open+0x356/0xb40 [ 5988.349899] do_last+0x313/0xfc0 [ 5988.354527] ? nd_jump_root+0xe5/0x160 [ 5988.361952] ? path_init+0x437/0x520 [ 5988.366824] path_openat+0xf7/0x500 [ 5988.371156] ? do_raw_spin_unlock+0x75/0x190 [ 5988.372756] do_filp_open+0x99/0x140 [ 5988.383613] ? getname_flags+0x6e/0x330 [ 5988.385019] ? __check_object_size+0xff/0x256 [ 5988.402121] ? do_raw_spin_unlock+0x75/0x190 [ 5988.404799] ? _raw_spin_unlock+0x12/0x30 [ 5988.414375] do_sys_openat2+0x2b4/0x410 [ 5988.418461] do_sys_open+0x73/0xa0 [ 5988.421303] __x64_sys_openat+0x24/0x30 [ 5988.424540] do_syscall_64+0xc1/0x440 [ 5988.427613] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5988.429530] RIP: 0033:0x7f1a63983332 [ 5988.442236] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5988.459910] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5988.472771] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 5988.481968] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 5988.491714] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 5988.500486] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 5988.511090] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 5991.400510] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3968s! [ 5991.407858] Showing busy workqueues and worker pools: [ 5991.410750] workqueue events: flags=0x0 [ 5991.412225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5991.420330] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 5991.426359] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5991.435479] pending: sec_gc_main [ptlrpc] [ 5991.439026] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5991.443074] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 5991.451506] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5991.454962] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 5991.467014] workqueue events_long: flags=0x0 [ 5991.469674] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5991.483752] pending: lu_objects_flush [obdclass] [ 5991.492397] workqueue events_unbound: flags=0x2 [ 5991.499293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5991.516086] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5991.546424] workqueue events_power_efficient: flags=0x80 [ 5991.548443] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5991.562544] pending: neigh_periodic_work, do_cache_clean [ 5991.581022] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5991.592697] pending: neigh_periodic_work, check_lifetime [ 5991.602356] workqueue mm_percpu_wq: flags=0x8 [ 5991.609700] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5991.626213] pending: vmstat_update [ 5991.627690] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5991.648078] pending: vmstat_update [ 5991.651479] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5991.655587] pending: vmstat_update [ 5991.658403] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5991.662493] pending: vmstat_update [ 5991.665062] workqueue writeback: flags=0x4a [ 5991.667536] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5991.671423] pending: wb_workfn, wb_workfn [ 5991.674310] workqueue xprtiod: flags=0xa [ 5991.677040] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5991.681543] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 5991.686435] workqueue ipv6_addrconf: flags=0x40008 [ 5991.689292] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5991.693147] pending: addrconf_verify_work [ 5991.696054] workqueue knbd0-recv: flags=0x1a [ 5991.698779] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5991.702958] in-flight: 647:recv_work [ 5991.705165] workqueue ptlrpc_pinger: flags=0x2 [ 5991.707462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5991.710420] pending: ptlrpc_pinger_main [ptlrpc] [ 5991.713073] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5981s workers=2 idle: 652 [ 6000.056031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6000.061571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6000.070609] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6000.104929] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6000.104941] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6000.104943] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6000.104957] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6000.104961] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6000.104967] RAX: 0000000000001304 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6000.104970] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6000.104972] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6000.104974] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6000.104977] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6000.104986] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6000.104989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6000.104992] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6000.105001] Call Trace: [ 6000.105007] [ 6000.105012] ? show_regs.cold.9+0x22/0x2f [ 6000.105022] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6000.105029] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6000.105033] ? watchdog+0x40/0x40 [ 6000.105036] ? hrtimer_interrupt+0x128/0x2c0 [ 6000.105040] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6000.105047] ? apic_timer_interrupt+0xf/0x20 [ 6000.105051] [ 6000.105053] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6000.105059] do_raw_spin_lock+0x177/0x190 [ 6000.105070] _raw_spin_lock+0x21/0x30 [ 6000.105075] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6000.105713] ? do_raw_spin_unlock+0x75/0x190 [ 6000.105739] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6000.105940] ? do_wait_intr+0xf0/0xf0 [ 6000.105948] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6000.106137] mdc_close+0x53b/0xd70 [mdc] [ 6000.106174] lmv_close+0x18d/0x490 [lmv] [ 6000.106206] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6000.106437] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6000.106520] ll_md_close+0x274/0x800 [lustre] [ 6000.106599] ll_file_release+0x128/0xc50 [lustre] [ 6000.106676] __fput+0x10d/0x440 [ 6000.106685] ____fput+0x12/0x20 [ 6000.106689] task_work_run+0xc6/0x110 [ 6000.106695] exit_to_usermode_loop+0x1dd/0x1f0 [ 6000.106702] do_syscall_64+0x3d6/0x440 [ 6000.106706] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6000.106712] RIP: 0033:0x7f1a63983cd8 [ 6000.106788] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6000.106794] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6000.106799] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6000.106802] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6000.106804] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6000.106807] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6000.106809] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6000.353636] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6000.371300] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6000.378960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6000.395822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6000.405654] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6000.422757] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6000.427611] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6000.435554] RDX: 0000000000000bd2 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6000.441907] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6000.454537] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6000.456715] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6000.471023] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6000.483912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6000.486875] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6000.490672] Call Trace: [ 6000.492030] [ 6000.493702] ? show_regs.cold.9+0x22/0x2f [ 6000.495513] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6000.498164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6000.500329] ? watchdog+0x40/0x40 [ 6000.502653] ? hrtimer_interrupt+0x128/0x2c0 [ 6000.504662] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6000.507163] ? apic_timer_interrupt+0xf/0x20 [ 6000.509541] [ 6000.510552] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6000.513574] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6000.516387] do_raw_spin_lock+0x177/0x190 [ 6000.518595] _raw_spin_lock+0x21/0x30 [ 6000.520538] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6000.523507] obd_mod_free+0x46/0x140 [ptlrpc] [ 6000.526320] mdc_commit_open+0x91/0xb0 [mdc] [ 6000.528636] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6000.531631] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6000.534765] after_reply+0xe2c/0x1780 [ptlrpc] [ 6000.537620] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6000.540638] ? __next_timer_interrupt+0x160/0x160 [ 6000.543084] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6000.546005] ? do_wait_intr+0xf0/0xf0 [ 6000.547977] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6000.550995] kthread+0x1d1/0x200 [ 6000.553034] ? set_kthread_struct+0x70/0x70 [ 6000.555558] ret_from_fork+0x1f/0x30 [ 6004.057049] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6004.062420] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6004.087187] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6004.094029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6004.099462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6004.102131] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6004.112595] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6004.117309] RAX: 000000000000054a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6004.121326] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6004.123652] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6004.126891] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6004.131448] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6004.134682] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6004.138290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6004.141367] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6004.145191] Call Trace: [ 6004.146769] [ 6004.147842] ? show_regs.cold.9+0x22/0x2f [ 6004.149948] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6004.152339] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6004.154648] ? watchdog+0x40/0x40 [ 6004.156667] ? hrtimer_interrupt+0x128/0x2c0 [ 6004.158838] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6004.161567] ? apic_timer_interrupt+0xf/0x20 [ 6004.163666] [ 6004.164521] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6004.167201] do_raw_spin_lock+0x177/0x190 [ 6004.169264] _raw_spin_lock+0x21/0x30 [ 6004.171008] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6004.173516] ? do_raw_spin_unlock+0x75/0x190 [ 6004.175488] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6004.178131] ? do_wait_intr+0xf0/0xf0 [ 6004.179633] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6004.182165] mdc_close+0x53b/0xd70 [mdc] [ 6004.184223] lmv_close+0x18d/0x490 [lmv] [ 6004.186240] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6004.188669] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6004.191020] ll_md_close+0x274/0x800 [lustre] [ 6004.193031] ll_file_release+0x128/0xc50 [lustre] [ 6004.195851] __fput+0x10d/0x440 [ 6004.197493] ____fput+0x12/0x20 [ 6004.198958] task_work_run+0xc6/0x110 [ 6004.200990] exit_to_usermode_loop+0x1dd/0x1f0 [ 6004.202891] do_syscall_64+0x3d6/0x440 [ 6004.204863] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6004.207146] RIP: 0033:0x7f1a63983cd8 [ 6004.208880] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6004.218264] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6004.221828] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6004.225041] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6004.228677] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6004.232311] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6004.235465] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6016.053051] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6016.055449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6016.103180] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6016.110156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6016.116145] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6016.119746] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6016.132409] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6016.137121] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6016.142071] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6016.145375] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6016.150193] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6016.155258] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6016.159093] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6016.164648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6016.168208] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6016.173458] Call Trace: [ 6016.174990] [ 6016.176527] ? show_regs.cold.9+0x22/0x2f [ 6016.179025] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6016.182187] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6016.184771] ? watchdog+0x40/0x40 [ 6016.186871] ? hrtimer_interrupt+0x128/0x2c0 [ 6016.190259] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6016.193153] ? apic_timer_interrupt+0xf/0x20 [ 6016.195955] [ 6016.197413] ? native_safe_halt+0x2e/0x50 [ 6016.199995] kvm_wait+0xcd/0xe0 [ 6016.203745] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6016.207543] do_raw_spin_lock+0x177/0x190 [ 6016.211183] _raw_spin_lock+0x21/0x30 [ 6016.214103] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6016.218290] ? do_raw_spin_unlock+0x75/0x190 [ 6016.221316] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6016.226788] ? do_wait_intr+0xf0/0xf0 [ 6016.229521] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6016.233677] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6016.237549] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6016.245676] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.250078] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6016.255597] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6016.261549] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6016.265697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.270664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6016.277004] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6016.279104] lmv_intent_open+0x338/0x1150 [lmv] [ 6016.287018] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.291957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.295874] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6016.301901] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6016.309288] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.320206] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6016.324002] ? do_raw_spin_unlock+0x75/0x190 [ 6016.327781] ? _raw_spin_unlock+0x12/0x30 [ 6016.330153] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6016.335525] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6016.339926] ll_atomic_open+0x36a/0xd70 [lustre] [ 6016.344723] lookup_open+0x356/0xb40 [ 6016.347453] do_last+0x313/0xfc0 [ 6016.349687] ? nd_jump_root+0xe5/0x160 [ 6016.352709] ? path_init+0x437/0x520 [ 6016.353994] path_openat+0xf7/0x500 [ 6016.356681] ? do_raw_spin_unlock+0x75/0x190 [ 6016.359066] do_filp_open+0x99/0x140 [ 6016.361765] ? getname_flags+0x6e/0x330 [ 6016.364750] ? __check_object_size+0xff/0x256 [ 6016.367807] ? do_raw_spin_unlock+0x75/0x190 [ 6016.370996] ? _raw_spin_unlock+0x12/0x30 [ 6016.376792] do_sys_openat2+0x2b4/0x410 [ 6016.379797] do_sys_open+0x73/0xa0 [ 6016.383247] __x64_sys_openat+0x24/0x30 [ 6016.385945] do_syscall_64+0xc1/0x440 [ 6016.388937] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6016.392800] RIP: 0033:0x7f1a63983332 [ 6016.395923] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6016.408025] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6016.413743] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6016.420915] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6016.426849] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6016.436045] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6016.444199] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6022.113113] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3999s! [ 6022.116656] Showing busy workqueues and worker pools: [ 6022.123280] workqueue events: flags=0x0 [ 6022.126710] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6022.134337] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6022.142807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6022.145920] pending: sec_gc_main [ptlrpc] [ 6022.153258] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6022.160384] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6022.168867] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6022.173895] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6022.181859] workqueue events_long: flags=0x0 [ 6022.183892] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6022.189264] pending: lu_objects_flush [obdclass] [ 6022.191787] workqueue events_unbound: flags=0x2 [ 6022.198815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6022.205722] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6022.212501] workqueue events_power_efficient: flags=0x80 [ 6022.214836] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6022.217646] pending: neigh_periodic_work, do_cache_clean [ 6022.219863] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6022.223751] pending: neigh_periodic_work, check_lifetime [ 6022.226763] workqueue mm_percpu_wq: flags=0x8 [ 6022.228614] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6022.232697] pending: vmstat_update [ 6022.234093] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6022.237254] pending: vmstat_update [ 6022.238600] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6022.241668] pending: vmstat_update [ 6022.242963] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6022.245213] pending: vmstat_update [ 6022.246384] workqueue writeback: flags=0x4a [ 6022.248342] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6022.250805] pending: wb_workfn, wb_workfn [ 6022.252504] workqueue xprtiod: flags=0xa [ 6022.254690] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6022.262361] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6022.272966] workqueue ipv6_addrconf: flags=0x40008 [ 6022.277043] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6022.285095] pending: addrconf_verify_work [ 6022.287859] workqueue knbd0-recv: flags=0x1a [ 6022.289830] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6022.294996] in-flight: 647:recv_work [ 6022.297163] workqueue ptlrpc_pinger: flags=0x2 [ 6022.299873] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6022.305484] pending: ptlrpc_pinger_main [ptlrpc] [ 6022.308154] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6012s workers=2 idle: 652 [ 6028.056019] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 6028.058006] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6028.058204] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6028.058274] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6028.058279] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6028.058282] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6028.058293] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6028.058297] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6028.058301] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6028.058303] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6028.058305] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6028.058307] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6028.058310] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6028.058313] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6028.058316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6028.058318] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6028.058326] Call Trace: [ 6028.058330] [ 6028.058335] ? show_regs.cold.9+0x22/0x2f [ 6028.058343] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6028.058349] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6028.058353] ? watchdog+0x40/0x40 [ 6028.058356] ? hrtimer_interrupt+0x128/0x2c0 [ 6028.063311] Modules linked in: [ 6028.075668] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6028.075686] ? apic_timer_interrupt+0xf/0x20 [ 6028.075689] [ 6028.075693] ? native_safe_halt+0x2e/0x50 [ 6028.075696] kvm_wait+0xcd/0xe0 [ 6028.075702] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6028.075708] do_raw_spin_lock+0x177/0x190 [ 6028.075711] _raw_spin_lock+0x21/0x30 [ 6028.075714] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6028.090855] lustre(O) [ 6028.093239] ? do_raw_spin_unlock+0x75/0x190 [ 6028.099420] osp(O) [ 6028.124820] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6028.125101] ? do_wait_intr+0xf0/0xf0 [ 6028.125114] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6028.132107] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6028.132200] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6028.132208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6028.132211] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6028.132227] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6028.132231] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6028.132236] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6028.132239] RDX: 0000000000001576 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6028.132242] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6028.132244] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6028.132247] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6028.132251] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6028.132254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6028.132256] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6028.132266] Call Trace: [ 6028.132270] [ 6028.132275] ? show_regs.cold.9+0x22/0x2f [ 6028.132284] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6028.132291] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6028.132296] ? watchdog+0x40/0x40 [ 6028.132301] ? hrtimer_interrupt+0x128/0x2c0 [ 6028.132305] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6028.132313] ? apic_timer_interrupt+0xf/0x20 [ 6028.132319] [ 6028.132322] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6028.132327] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6028.132332] do_raw_spin_lock+0x177/0x190 [ 6028.132337] _raw_spin_lock+0x21/0x30 [ 6028.132343] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6028.132663] obd_mod_free+0x46/0x140 [ptlrpc] [ 6028.133543] mdc_commit_open+0x91/0xb0 [mdc] [ 6028.133591] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6028.133932] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6028.134287] after_reply+0xe2c/0x1780 [ptlrpc] [ 6028.134551] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6028.134835] ? __next_timer_interrupt+0x160/0x160 [ 6028.134845] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6028.135074] ? do_wait_intr+0xf0/0xf0 [ 6028.135082] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6028.135445] kthread+0x1d1/0x200 [ 6028.135645] ? set_kthread_struct+0x70/0x70 [ 6028.143402] mdc_close+0x53b/0xd70 [mdc] [ 6028.143537] lmv_close+0x18d/0x490 [lmv] [ 6028.143570] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6028.143706] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6028.143793] ll_md_close+0x274/0x800 [lustre] [ 6028.143892] ll_file_release+0x128/0xc50 [lustre] [ 6028.144070] __fput+0x10d/0x440 [ 6028.144080] ____fput+0x12/0x20 [ 6028.144084] task_work_run+0xc6/0x110 [ 6028.144092] exit_to_usermode_loop+0x1dd/0x1f0 [ 6028.144099] do_syscall_64+0x3d6/0x440 [ 6028.144103] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6028.144111] RIP: 0033:0x7f1a63983cd8 [ 6028.144117] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6028.144121] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6028.144139] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6028.144142] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6028.144144] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6028.144147] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6028.144149] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6028.353276] ret_from_fork+0x1f/0x30 [ 6032.057010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6032.067192] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6032.130149] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6032.141910] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6032.158045] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6032.164561] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6032.186371] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6032.201852] RAX: 0000000000007b2b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6032.205981] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6032.212237] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6032.223349] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6032.229294] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6032.231904] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6032.239225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6032.242305] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6032.250128] Call Trace: [ 6032.251489] [ 6032.254600] ? show_regs.cold.9+0x22/0x2f [ 6032.256740] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6032.262456] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6032.268460] ? watchdog+0x40/0x40 [ 6032.270605] ? hrtimer_interrupt+0x128/0x2c0 [ 6032.275909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6032.279877] ? apic_timer_interrupt+0xf/0x20 [ 6032.282761] [ 6032.285757] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6032.288703] do_raw_spin_lock+0x177/0x190 [ 6032.291957] _raw_spin_lock+0x21/0x30 [ 6032.296389] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6032.304213] ? do_raw_spin_unlock+0x75/0x190 [ 6032.306443] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6032.312358] ? do_wait_intr+0xf0/0xf0 [ 6032.314810] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6032.324794] mdc_close+0x53b/0xd70 [mdc] [ 6032.330794] lmv_close+0x18d/0x490 [lmv] [ 6032.336536] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6032.341549] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6032.346046] ll_md_close+0x274/0x800 [lustre] [ 6032.353964] ll_file_release+0x128/0xc50 [lustre] [ 6032.358854] __fput+0x10d/0x440 [ 6032.360628] ____fput+0x12/0x20 [ 6032.362189] task_work_run+0xc6/0x110 [ 6032.369116] exit_to_usermode_loop+0x1dd/0x1f0 [ 6032.376274] do_syscall_64+0x3d6/0x440 [ 6032.379574] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6032.380976] RIP: 0033:0x7f1a63983cd8 [ 6032.384799] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6032.404804] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6032.416165] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6032.423378] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6032.428426] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6032.433358] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6032.437527] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6042.498191] rcu: INFO: rcu_sched self-detected stall on CPU [ 6042.500265] rcu: 3-...!: (2458835 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 6042.503887] (t=4020097 jiffies g=269633 q=1621) [ 6042.505462] rcu: rcu_sched kthread starved for 4019745 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 6042.508659] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6042.511721] rcu: RCU grace-period kthread stack dump: [ 6042.513479] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6042.516479] Call Trace: [ 6042.517221] __schedule+0x351/0xcb0 [ 6042.518423] schedule+0xc0/0x180 [ 6042.519465] schedule_timeout+0xb4/0x190 [ 6042.520580] ? do_raw_spin_unlock+0x75/0x190 [ 6042.521802] ? __next_timer_interrupt+0x160/0x160 [ 6042.523504] ? prepare_to_swait_event+0xf3/0x140 [ 6042.524908] rcu_gp_kthread+0x8dd/0x15b0 [ 6042.526422] ? force_qs_rnp+0x2c0/0x2c0 [ 6042.528518] kthread+0x1d1/0x200 [ 6042.529540] ? set_kthread_struct+0x70/0x70 [ 6042.530772] ret_from_fork+0x1f/0x30 [ 6042.531798] rcu: Stack dump where RCU GP kthread last ran: [ 6042.534146] Sending NMI from CPU 3 to CPUs 2: [ 6042.536928] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 6042.539233] NMI backtrace for cpu 3 [ 6042.540407] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6042.543943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6042.546774] Call Trace: [ 6042.547606] [ 6042.548380] ? dump_stack+0xbb/0x10e [ 6042.549505] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6042.550882] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6042.552540] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6042.554183] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6042.555952] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6042.557617] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6042.558956] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6042.560895] ? update_process_times+0x6d/0xc0 [ 6042.562485] ? tick_sched_handle.isra.18+0x27/0x90 [ 6042.563921] ? tick_sched_timer+0x53/0xb0 [ 6042.565295] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6042.566688] ? tick_sched_handle.isra.18+0x90/0x90 [ 6042.568222] ? hrtimer_interrupt+0x128/0x2c0 [ 6042.569554] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6042.571118] ? apic_timer_interrupt+0xf/0x20 [ 6042.572587] [ 6042.573380] ? native_safe_halt+0x2e/0x50 [ 6042.574642] ? kvm_wait+0xcd/0xe0 [ 6042.575699] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6042.577499] ? do_raw_spin_lock+0x177/0x190 [ 6042.578805] ? _raw_spin_lock+0x21/0x30 [ 6042.580141] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6042.581939] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6042.583700] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6042.585213] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6042.587086] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6042.588990] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6042.590785] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6042.592685] ? __next_timer_interrupt+0x160/0x160 [ 6042.594070] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6042.595699] ? do_wait_intr+0xf0/0xf0 [ 6042.596807] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6042.598607] ? kthread+0x1d1/0x200 [ 6042.599668] ? set_kthread_struct+0x70/0x70 [ 6042.600939] ? ret_from_fork+0x1f/0x30 [ 6048.053216] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6048.060251] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6048.094116] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6048.103782] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6048.107803] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6048.112623] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6048.123673] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6048.137176] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6048.143200] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6048.151637] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6048.156220] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6048.166609] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6048.174392] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6048.179425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6048.182859] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6048.189543] Call Trace: [ 6048.190786] [ 6048.192941] ? show_regs.cold.9+0x22/0x2f [ 6048.196126] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6048.198854] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6048.202383] ? watchdog+0x40/0x40 [ 6048.203641] ? hrtimer_interrupt+0x128/0x2c0 [ 6048.215507] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6048.227309] ? apic_timer_interrupt+0xf/0x20 [ 6048.231373] [ 6048.233183] ? native_safe_halt+0x2e/0x50 [ 6048.238191] kvm_wait+0xcd/0xe0 [ 6048.239411] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6048.249685] do_raw_spin_lock+0x177/0x190 [ 6048.262689] _raw_spin_lock+0x21/0x30 [ 6048.267176] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6048.269250] ? do_raw_spin_unlock+0x75/0x190 [ 6048.276140] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6048.292863] ? do_wait_intr+0xf0/0xf0 [ 6048.294298] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6048.299792] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6048.304695] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6048.307118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6048.311915] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6048.317311] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6048.321545] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6048.323734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6048.325812] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6048.329865] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6048.339184] lmv_intent_open+0x338/0x1150 [lmv] [ 6048.346126] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6048.352732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6048.355850] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6048.359008] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6048.365370] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6048.367467] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6048.377684] ? do_raw_spin_unlock+0x75/0x190 [ 6048.381757] ? _raw_spin_unlock+0x12/0x30 [ 6048.387639] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6048.395194] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6048.397142] ll_atomic_open+0x36a/0xd70 [lustre] [ 6048.399630] lookup_open+0x356/0xb40 [ 6048.401032] do_last+0x313/0xfc0 [ 6048.403627] ? nd_jump_root+0xe5/0x160 [ 6048.406701] ? path_init+0x437/0x520 [ 6048.410437] path_openat+0xf7/0x500 [ 6048.412865] ? do_raw_spin_unlock+0x75/0x190 [ 6048.416213] do_filp_open+0x99/0x140 [ 6048.419632] ? getname_flags+0x6e/0x330 [ 6048.421935] ? __check_object_size+0xff/0x256 [ 6048.424680] ? do_raw_spin_unlock+0x75/0x190 [ 6048.427439] ? _raw_spin_unlock+0x12/0x30 [ 6048.429914] do_sys_openat2+0x2b4/0x410 [ 6048.432252] do_sys_open+0x73/0xa0 [ 6048.433422] __x64_sys_openat+0x24/0x30 [ 6048.435865] do_syscall_64+0xc1/0x440 [ 6048.437155] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6048.440581] RIP: 0033:0x7f1a63983332 [ 6048.443093] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6048.455948] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6048.460956] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6048.464975] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6048.467998] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6048.479345] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6048.484863] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6052.322437] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4029s! [ 6052.336997] Showing busy workqueues and worker pools: [ 6052.342734] workqueue events: flags=0x0 [ 6052.350468] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6052.357306] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6052.364545] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6052.382486] pending: sec_gc_main [ptlrpc] [ 6052.392810] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6052.397667] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6052.403281] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6052.408501] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6052.418236] workqueue events_long: flags=0x0 [ 6052.422767] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6052.426686] pending: lu_objects_flush [obdclass] [ 6052.429321] workqueue events_unbound: flags=0x2 [ 6052.431956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6052.435500] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6052.441422] workqueue events_power_efficient: flags=0x80 [ 6052.455639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6052.461748] pending: neigh_periodic_work, do_cache_clean [ 6052.465705] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6052.471590] pending: neigh_periodic_work, check_lifetime [ 6052.475597] workqueue mm_percpu_wq: flags=0x8 [ 6052.477590] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6052.480591] pending: vmstat_update [ 6052.482654] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6052.494190] pending: vmstat_update [ 6052.496738] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6052.505656] pending: vmstat_update [ 6052.508664] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6052.512533] pending: vmstat_update [ 6052.515959] workqueue writeback: flags=0x4a [ 6052.524668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6052.536618] pending: wb_workfn, wb_workfn [ 6052.541746] workqueue xprtiod: flags=0xa [ 6052.548098] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6052.554558] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6052.567762] workqueue ipv6_addrconf: flags=0x40008 [ 6052.578329] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6052.583045] pending: addrconf_verify_work [ 6052.588652] workqueue knbd0-recv: flags=0x1a [ 6052.590159] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6052.596533] in-flight: 647:recv_work [ 6052.599031] workqueue ptlrpc_pinger: flags=0x2 [ 6052.601814] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6052.609396] pending: ptlrpc_pinger_main [ptlrpc] [ 6052.619523] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6042s workers=2 idle: 652 [ 6056.058289] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 6056.078551] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6056.129539] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6056.136267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6056.140542] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6056.143195] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6056.153839] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6056.161105] RAX: 0000000000005a91 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6056.169973] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6056.179740] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6056.186413] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6056.191919] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6056.199514] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6056.208428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6056.214845] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6056.222518] Call Trace: [ 6056.223733] [ 6056.226616] ? show_regs.cold.9+0x22/0x2f [ 6056.228351] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6056.231042] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6056.237329] ? watchdog+0x40/0x40 [ 6056.239682] ? hrtimer_interrupt+0x128/0x2c0 [ 6056.241617] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6056.249673] ? apic_timer_interrupt+0xf/0x20 [ 6056.255600] [ 6056.256303] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6056.260686] do_raw_spin_lock+0x177/0x190 [ 6056.261826] _raw_spin_lock+0x21/0x30 [ 6056.263041] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6056.268690] ? do_raw_spin_unlock+0x75/0x190 [ 6056.270167] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6056.273930] ? do_wait_intr+0xf0/0xf0 [ 6056.277956] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6056.282703] mdc_close+0x53b/0xd70 [mdc] [ 6056.284470] lmv_close+0x18d/0x490 [lmv] [ 6056.286687] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6056.290949] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6056.295543] ll_md_close+0x274/0x800 [lustre] [ 6056.298551] ll_file_release+0x128/0xc50 [lustre] [ 6056.300981] __fput+0x10d/0x440 [ 6056.302085] ____fput+0x12/0x20 [ 6056.303965] task_work_run+0xc6/0x110 [ 6056.306114] exit_to_usermode_loop+0x1dd/0x1f0 [ 6056.311753] do_syscall_64+0x3d6/0x440 [ 6056.313947] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6056.319567] RIP: 0033:0x7f1a63983cd8 [ 6056.321626] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6056.332546] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6056.343807] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6056.349783] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6056.357584] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6056.363578] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6056.371613] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6068.057017] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 6068.060753] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6068.060994] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6068.061143] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6068.061150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6068.061153] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6068.061166] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6068.061170] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6068.061175] RAX: 0000000000001149 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6068.061177] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6068.061179] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6068.061182] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6068.061265] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6068.061390] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6068.061395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6068.061398] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6068.061405] Call Trace: [ 6068.061411] [ 6068.061417] ? show_regs.cold.9+0x22/0x2f [ 6068.061427] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6068.061433] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6068.061438] ? watchdog+0x40/0x40 [ 6068.061442] ? hrtimer_interrupt+0x128/0x2c0 [ 6068.061447] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6068.061455] ? apic_timer_interrupt+0xf/0x20 [ 6068.061460] [ 6068.061463] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6068.061470] do_raw_spin_lock+0x177/0x190 [ 6068.061475] _raw_spin_lock+0x21/0x30 [ 6068.061481] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6068.061958] ? do_raw_spin_unlock+0x75/0x190 [ 6068.061966] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6068.062221] ? do_wait_intr+0xf0/0xf0 [ 6068.062231] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6068.062629] mdc_close+0x53b/0xd70 [mdc] [ 6068.062671] lmv_close+0x18d/0x490 [lmv] [ 6068.062703] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6068.062900] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6068.062989] ll_md_close+0x274/0x800 [lustre] [ 6068.063096] ll_file_release+0x128/0xc50 [lustre] [ 6068.063237] __fput+0x10d/0x440 [ 6068.063252] ____fput+0x12/0x20 [ 6068.063260] task_work_run+0xc6/0x110 [ 6068.063268] exit_to_usermode_loop+0x1dd/0x1f0 [ 6068.063287] do_syscall_64+0x3d6/0x440 [ 6068.063292] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6068.063300] RIP: 0033:0x7f1a63983cd8 [ 6068.063352] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6068.063359] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6068.063369] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6068.063372] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6068.063378] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6068.063381] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6068.063383] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6068.356883] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6068.389195] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6068.394602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6068.402452] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6068.413692] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6068.426731] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6068.433842] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6068.440888] RDX: 0000000000006266 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6068.446895] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6068.453362] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6068.458423] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6068.464580] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6068.470006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6068.474039] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6068.478660] Call Trace: [ 6068.481346] [ 6068.482597] ? show_regs.cold.9+0x22/0x2f [ 6068.486205] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6068.489950] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6068.492628] ? watchdog+0x40/0x40 [ 6068.495014] ? hrtimer_interrupt+0x128/0x2c0 [ 6068.498828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6068.501659] ? apic_timer_interrupt+0xf/0x20 [ 6068.503043] [ 6068.503816] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6068.507921] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6068.513146] do_raw_spin_lock+0x177/0x190 [ 6068.516630] _raw_spin_lock+0x21/0x30 [ 6068.519349] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6068.524221] obd_mod_free+0x46/0x140 [ptlrpc] [ 6068.528850] mdc_commit_open+0x91/0xb0 [mdc] [ 6068.530701] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6068.536030] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6068.540157] after_reply+0xe2c/0x1780 [ptlrpc] [ 6068.545526] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6068.547481] ? __next_timer_interrupt+0x160/0x160 [ 6068.554757] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6068.556810] ? do_wait_intr+0xf0/0xf0 [ 6068.562472] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6068.564718] kthread+0x1d1/0x200 [ 6068.568310] ? set_kthread_struct+0x70/0x70 [ 6068.570933] ret_from_fork+0x1f/0x30 [ 6080.054070] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6080.058511] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6080.093068] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6080.102280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6080.112202] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6080.117663] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6080.136692] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6080.147216] RAX: 0000000000006613 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6080.154495] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6080.160804] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6080.168689] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6080.175606] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6080.184235] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6080.189436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6080.195743] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6080.198889] Call Trace: [ 6080.199795] [ 6080.203139] ? show_regs.cold.9+0x22/0x2f [ 6080.205554] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6080.214577] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6080.222349] ? watchdog+0x40/0x40 [ 6080.223535] ? hrtimer_interrupt+0x128/0x2c0 [ 6080.228607] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6080.236149] ? apic_timer_interrupt+0xf/0x20 [ 6080.240543] [ 6080.244165] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6080.251792] do_raw_spin_lock+0x177/0x190 [ 6080.252987] _raw_spin_lock+0x21/0x30 [ 6080.254092] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6080.267433] ? do_raw_spin_unlock+0x75/0x190 [ 6080.276590] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6080.288917] ? do_wait_intr+0xf0/0xf0 [ 6080.292547] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6080.298334] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6080.301894] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6080.305155] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6080.311178] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6080.317458] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6080.321663] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6080.326942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6080.332722] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6080.340189] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6080.346542] lmv_intent_open+0x338/0x1150 [lmv] [ 6080.351496] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6080.356791] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6080.362411] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6080.368200] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6080.374153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6080.378475] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6080.383777] ? do_raw_spin_unlock+0x75/0x190 [ 6080.386865] ? _raw_spin_unlock+0x12/0x30 [ 6080.392315] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6080.398567] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6080.401164] ll_atomic_open+0x36a/0xd70 [lustre] [ 6080.405649] lookup_open+0x356/0xb40 [ 6080.408093] do_last+0x313/0xfc0 [ 6080.409679] ? nd_jump_root+0xe5/0x160 [ 6080.412335] ? path_init+0x437/0x520 [ 6080.416242] path_openat+0xf7/0x500 [ 6080.420116] ? do_raw_spin_unlock+0x75/0x190 [ 6080.427304] do_filp_open+0x99/0x140 [ 6080.431027] ? getname_flags+0x6e/0x330 [ 6080.433801] ? __check_object_size+0xff/0x256 [ 6080.437503] ? do_raw_spin_unlock+0x75/0x190 [ 6080.443235] ? _raw_spin_unlock+0x12/0x30 [ 6080.445904] do_sys_openat2+0x2b4/0x410 [ 6080.449680] do_sys_open+0x73/0xa0 [ 6080.452132] __x64_sys_openat+0x24/0x30 [ 6080.453589] do_syscall_64+0xc1/0x440 [ 6080.457495] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6080.461420] RIP: 0033:0x7f1a63983332 [ 6080.465287] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6080.484518] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6080.498553] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6080.505344] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6080.514654] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6080.522437] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6080.526541] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6083.044609] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4060s! [ 6083.059315] Showing busy workqueues and worker pools: [ 6083.066212] workqueue events: flags=0x0 [ 6083.067688] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6083.070597] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6083.076174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6083.081370] pending: sec_gc_main [ptlrpc] [ 6083.090914] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6083.096772] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6083.106539] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6083.116378] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6083.128454] workqueue events_long: flags=0x0 [ 6083.130078] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6083.141567] pending: lu_objects_flush [obdclass] [ 6083.148396] workqueue events_unbound: flags=0x2 [ 6083.156314] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6083.158597] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6083.170212] workqueue events_power_efficient: flags=0x80 [ 6083.173515] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6083.177491] pending: neigh_periodic_work, do_cache_clean [ 6083.183819] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6083.195599] pending: neigh_periodic_work, check_lifetime [ 6083.201513] workqueue mm_percpu_wq: flags=0x8 [ 6083.204319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6083.217528] pending: vmstat_update [ 6083.221163] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6083.228536] pending: vmstat_update [ 6083.234433] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6083.239448] pending: vmstat_update [ 6083.242061] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6083.252710] pending: vmstat_update [ 6083.257063] workqueue writeback: flags=0x4a [ 6083.261424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6083.269808] pending: wb_workfn, wb_workfn [ 6083.272914] workqueue xprtiod: flags=0xa [ 6083.275122] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6083.282074] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6083.285734] workqueue ipv6_addrconf: flags=0x40008 [ 6083.287717] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6083.290467] pending: addrconf_verify_work [ 6083.297102] workqueue knbd0-recv: flags=0x1a [ 6083.299786] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6083.308425] in-flight: 647:recv_work [ 6083.311388] workqueue ptlrpc_pinger: flags=0x2 [ 6083.316087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6083.319460] pending: ptlrpc_pinger_main [ptlrpc] [ 6083.325155] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6073s workers=2 idle: 652 [ 6084.056085] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 6084.058943] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6084.133268] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6084.147311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6084.156889] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6084.162289] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6084.179244] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6084.184500] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6084.188660] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6084.197701] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6084.209215] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6084.211763] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6084.216734] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6084.220881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6084.232028] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6084.234730] Call Trace: [ 6084.238507] [ 6084.240513] ? show_regs.cold.9+0x22/0x2f [ 6084.242469] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6084.246669] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6084.250474] ? watchdog+0x40/0x40 [ 6084.254508] ? hrtimer_interrupt+0x128/0x2c0 [ 6084.260617] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6084.268736] ? apic_timer_interrupt+0xf/0x20 [ 6084.278901] [ 6084.279459] ? native_safe_halt+0x2e/0x50 [ 6084.280470] kvm_wait+0xcd/0xe0 [ 6084.281384] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6084.292563] do_raw_spin_lock+0x177/0x190 [ 6084.297753] _raw_spin_lock+0x21/0x30 [ 6084.302284] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6084.309302] ? do_raw_spin_unlock+0x75/0x190 [ 6084.313706] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6084.321399] ? do_wait_intr+0xf0/0xf0 [ 6084.327012] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6084.333415] mdc_close+0x53b/0xd70 [mdc] [ 6084.339347] lmv_close+0x18d/0x490 [lmv] [ 6084.348991] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6084.353824] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6084.357664] ll_md_close+0x274/0x800 [lustre] [ 6084.364997] ll_file_release+0x128/0xc50 [lustre] [ 6084.378413] __fput+0x10d/0x440 [ 6084.383120] ____fput+0x12/0x20 [ 6084.387906] task_work_run+0xc6/0x110 [ 6084.392722] exit_to_usermode_loop+0x1dd/0x1f0 [ 6084.395698] do_syscall_64+0x3d6/0x440 [ 6084.402034] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6084.409422] RIP: 0033:0x7f1a63983cd8 [ 6084.415253] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6084.435057] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6084.437317] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6084.450214] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6084.460327] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6084.466901] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6084.473741] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6096.057116] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6096.059016] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6096.062674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6096.069498] Modules linked in: [ 6096.094640] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6096.094655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6096.094657] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6096.094672] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6096.094675] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6096.094681] RAX: 0000000000004d89 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6096.094683] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6096.094685] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6096.094688] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6096.094690] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6096.094693] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6096.094697] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6096.094699] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6096.094708] Call Trace: [ 6096.094714] [ 6096.094719] ? show_regs.cold.9+0x22/0x2f [ 6096.094729] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6096.094736] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6096.094739] ? watchdog+0x40/0x40 [ 6096.094744] ? hrtimer_interrupt+0x128/0x2c0 [ 6096.094748] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6096.094758] ? apic_timer_interrupt+0xf/0x20 [ 6096.094763] [ 6096.094765] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6096.094772] do_raw_spin_lock+0x177/0x190 [ 6096.094776] _raw_spin_lock+0x21/0x30 [ 6096.094781] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6096.095091] ? do_raw_spin_unlock+0x75/0x190 [ 6096.095097] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6096.095301] ? do_wait_intr+0xf0/0xf0 [ 6096.095309] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6096.095504] mdc_close+0x53b/0xd70 [mdc] [ 6096.095542] lmv_close+0x18d/0x490 [lmv] [ 6096.095573] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6096.095685] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6096.095768] ll_md_close+0x274/0x800 [lustre] [ 6096.095864] ll_file_release+0x128/0xc50 [lustre] [ 6096.095946] __fput+0x10d/0x440 [ 6096.095955] ____fput+0x12/0x20 [ 6096.095959] task_work_run+0xc6/0x110 [ 6096.095967] exit_to_usermode_loop+0x1dd/0x1f0 [ 6096.095974] do_syscall_64+0x3d6/0x440 [ 6096.095978] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6096.095984] RIP: 0033:0x7f1a63983cd8 [ 6096.095990] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6096.095992] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6096.095996] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6096.095999] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6096.096001] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6096.096003] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6096.096006] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6096.327288] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6096.367081] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6096.375458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6096.384164] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6096.387155] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6096.400368] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6096.404165] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6096.408118] RDX: 0000000000003b6c RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6096.411123] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6096.415252] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6096.418414] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6096.421580] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6096.426063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6096.428923] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6096.432252] Call Trace: [ 6096.433295] [ 6096.434201] ? show_regs.cold.9+0x22/0x2f [ 6096.436105] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6096.438141] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6096.441299] ? watchdog+0x40/0x40 [ 6096.442714] ? hrtimer_interrupt+0x128/0x2c0 [ 6096.444959] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6096.447573] ? apic_timer_interrupt+0xf/0x20 [ 6096.449565] [ 6096.450719] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6096.453251] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6096.455727] do_raw_spin_lock+0x177/0x190 [ 6096.463965] _raw_spin_lock+0x21/0x30 [ 6096.465416] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6096.471100] obd_mod_free+0x46/0x140 [ptlrpc] [ 6096.476471] mdc_commit_open+0x91/0xb0 [mdc] [ 6096.478287] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6096.483951] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6096.489678] after_reply+0xe2c/0x1780 [ptlrpc] [ 6096.495061] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6096.498468] ? __next_timer_interrupt+0x160/0x160 [ 6096.501944] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6096.503913] ? do_wait_intr+0xf0/0xf0 [ 6096.505618] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6096.509204] kthread+0x1d1/0x200 [ 6096.510762] ? set_kthread_struct+0x70/0x70 [ 6096.512053] ret_from_fork+0x1f/0x30 [ 6108.053090] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6108.058325] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6108.096028] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6108.100513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6108.104091] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6108.106007] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6108.113325] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6108.116512] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6108.119498] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6108.122474] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6108.125629] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6108.128867] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6108.131876] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6108.135216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6108.137740] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6108.140546] Call Trace: [ 6108.141646] [ 6108.142527] ? show_regs.cold.9+0x22/0x2f [ 6108.144217] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6108.146177] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6108.147835] ? watchdog+0x40/0x40 [ 6108.149040] ? hrtimer_interrupt+0x128/0x2c0 [ 6108.150607] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6108.152487] ? apic_timer_interrupt+0xf/0x20 [ 6108.154030] [ 6108.154877] ? native_safe_halt+0x2e/0x50 [ 6108.156273] kvm_wait+0xcd/0xe0 [ 6108.157473] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6108.159439] do_raw_spin_lock+0x177/0x190 [ 6108.160901] _raw_spin_lock+0x21/0x30 [ 6108.162220] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6108.164555] ? do_raw_spin_unlock+0x75/0x190 [ 6108.166084] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6108.168111] ? do_wait_intr+0xf0/0xf0 [ 6108.169461] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6108.171534] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6108.173616] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6108.176067] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6108.178205] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6108.180226] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6108.181887] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6108.183507] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6108.185452] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6108.187911] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6108.189917] lmv_intent_open+0x338/0x1150 [lmv] [ 6108.191550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6108.193644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6108.195644] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6108.197320] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6108.199225] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6108.201231] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6108.203001] ? do_raw_spin_unlock+0x75/0x190 [ 6108.204372] ? _raw_spin_unlock+0x12/0x30 [ 6108.205908] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6108.207945] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6108.209641] ll_atomic_open+0x36a/0xd70 [lustre] [ 6108.211527] lookup_open+0x356/0xb40 [ 6108.212808] do_last+0x313/0xfc0 [ 6108.213917] ? nd_jump_root+0xe5/0x160 [ 6108.215216] ? path_init+0x437/0x520 [ 6108.216538] path_openat+0xf7/0x500 [ 6108.217827] ? do_raw_spin_unlock+0x75/0x190 [ 6108.219627] do_filp_open+0x99/0x140 [ 6108.220848] ? getname_flags+0x6e/0x330 [ 6108.222157] ? __check_object_size+0xff/0x256 [ 6108.223816] ? do_raw_spin_unlock+0x75/0x190 [ 6108.225281] ? _raw_spin_unlock+0x12/0x30 [ 6108.226825] do_sys_openat2+0x2b4/0x410 [ 6108.228181] do_sys_open+0x73/0xa0 [ 6108.229497] __x64_sys_openat+0x24/0x30 [ 6108.230954] do_syscall_64+0xc1/0x440 [ 6108.232247] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6108.234084] RIP: 0033:0x7f1a63983332 [ 6108.235472] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6108.242483] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6108.245235] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6108.247823] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6108.250544] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6108.253156] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6108.255971] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6112.056262] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 6112.072504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6112.121055] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6112.130590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6112.139840] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6112.142816] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6112.158419] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6112.164761] RAX: 0000000000005d28 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6112.179054] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6112.194145] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6112.203963] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6112.210525] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6112.221986] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6112.237279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6112.248930] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6112.256379] Call Trace: [ 6112.257336] [ 6112.267564] ? show_regs.cold.9+0x22/0x2f [ 6112.270667] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6112.282276] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6112.291230] ? watchdog+0x40/0x40 [ 6112.292490] ? hrtimer_interrupt+0x128/0x2c0 [ 6112.293954] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6112.301402] ? apic_timer_interrupt+0xf/0x20 [ 6112.311818] [ 6112.313232] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6112.319408] do_raw_spin_lock+0x177/0x190 [ 6112.324958] _raw_spin_lock+0x21/0x30 [ 6112.326859] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6112.334530] ? do_raw_spin_unlock+0x75/0x190 [ 6112.337476] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6112.340690] ? do_wait_intr+0xf0/0xf0 [ 6112.342546] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6112.350261] mdc_close+0x53b/0xd70 [mdc] [ 6112.357318] lmv_close+0x18d/0x490 [lmv] [ 6112.362359] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6112.370651] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6112.376299] ll_md_close+0x274/0x800 [lustre] [ 6112.383446] ll_file_release+0x128/0xc50 [lustre] [ 6112.387476] __fput+0x10d/0x440 [ 6112.389324] ____fput+0x12/0x20 [ 6112.390593] task_work_run+0xc6/0x110 [ 6112.397552] exit_to_usermode_loop+0x1dd/0x1f0 [ 6112.402609] do_syscall_64+0x3d6/0x440 [ 6112.407002] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6112.408606] RIP: 0033:0x7f1a63983cd8 [ 6112.409580] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6112.432048] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6112.438507] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6112.449275] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6112.455877] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6112.465273] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6112.475247] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6113.764855] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4090s! [ 6113.779909] Showing busy workqueues and worker pools: [ 6113.785358] workqueue events: flags=0x0 [ 6113.790558] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6113.803056] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6113.810259] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6113.813829] pending: sec_gc_main [ptlrpc] [ 6113.815106] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6113.828043] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6113.850892] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6113.862754] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6113.884992] workqueue events_long: flags=0x0 [ 6113.890834] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6113.899772] pending: lu_objects_flush [obdclass] [ 6113.906141] workqueue events_unbound: flags=0x2 [ 6113.916126] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6113.924202] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6113.932884] workqueue events_power_efficient: flags=0x80 [ 6113.935140] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6113.944198] pending: neigh_periodic_work, do_cache_clean [ 6113.951240] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6113.961443] pending: neigh_periodic_work, check_lifetime [ 6113.969478] workqueue mm_percpu_wq: flags=0x8 [ 6113.971136] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6113.981317] pending: vmstat_update [ 6113.984857] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6113.995271] pending: vmstat_update [ 6114.006371] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6114.012870] pending: vmstat_update [ 6114.022183] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6114.042960] pending: vmstat_update [ 6114.048486] workqueue writeback: flags=0x4a [ 6114.062554] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6114.075365] pending: wb_workfn, wb_workfn [ 6114.085232] workqueue xprtiod: flags=0xa [ 6114.095605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6114.116023] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6114.131676] workqueue ipv6_addrconf: flags=0x40008 [ 6114.146041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6114.155447] pending: addrconf_verify_work [ 6114.164948] workqueue knbd0-recv: flags=0x1a [ 6114.177789] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6114.190789] in-flight: 647:recv_work [ 6114.200742] workqueue ptlrpc_pinger: flags=0x2 [ 6114.205090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6114.218819] pending: ptlrpc_pinger_main [ptlrpc] [ 6114.232883] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6104s workers=2 idle: 652 [ 6124.057872] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6124.058017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6124.061377] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6124.061578] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6124.061585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6124.061588] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6124.061603] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6124.061606] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6124.061611] RAX: 0000000000006f43 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6124.061613] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6124.061615] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6124.061617] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6124.061619] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6124.061623] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6124.061626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6124.061628] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6124.061636] Call Trace: [ 6124.061642] [ 6124.061647] ? show_regs.cold.9+0x22/0x2f [ 6124.061657] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6124.061663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6124.061667] ? watchdog+0x40/0x40 [ 6124.061671] ? hrtimer_interrupt+0x128/0x2c0 [ 6124.061676] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6124.061684] ? apic_timer_interrupt+0xf/0x20 [ 6124.061689] [ 6124.061692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6124.061697] do_raw_spin_lock+0x177/0x190 [ 6124.061702] _raw_spin_lock+0x21/0x30 [ 6124.061707] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6124.062047] ? do_raw_spin_unlock+0x75/0x190 [ 6124.062053] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6124.062798] ? do_wait_intr+0xf0/0xf0 [ 6124.062807] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6124.063032] mdc_close+0x53b/0xd70 [mdc] [ 6124.063074] lmv_close+0x18d/0x490 [lmv] [ 6124.063106] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6124.063218] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6124.063300] ll_md_close+0x274/0x800 [lustre] [ 6124.063382] ll_file_release+0x128/0xc50 [lustre] [ 6124.063691] __fput+0x10d/0x440 [ 6124.063699] ____fput+0x12/0x20 [ 6124.063703] task_work_run+0xc6/0x110 [ 6124.063709] exit_to_usermode_loop+0x1dd/0x1f0 [ 6124.063717] do_syscall_64+0x3d6/0x440 [ 6124.063721] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6124.063733] RIP: 0033:0x7f1a63983cd8 [ 6124.063740] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6124.063744] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6124.063752] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6124.063755] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6124.063757] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6124.063830] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6124.063833] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6124.295782] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6124.323174] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6124.328739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6124.336272] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6124.339743] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6124.350263] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6124.354471] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6124.358820] RDX: 000000000000787e RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6124.363227] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6124.367408] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6124.371741] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6124.375884] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6124.381000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6124.383721] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6124.388475] Call Trace: [ 6124.389918] [ 6124.390921] ? show_regs.cold.9+0x22/0x2f [ 6124.393489] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6124.396474] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6124.399791] ? watchdog+0x40/0x40 [ 6124.402184] ? hrtimer_interrupt+0x128/0x2c0 [ 6124.404677] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6124.407465] ? apic_timer_interrupt+0xf/0x20 [ 6124.409861] [ 6124.411252] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6124.414207] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6124.417224] do_raw_spin_lock+0x177/0x190 [ 6124.420064] _raw_spin_lock+0x21/0x30 [ 6124.422042] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6124.425329] obd_mod_free+0x46/0x140 [ptlrpc] [ 6124.428317] mdc_commit_open+0x91/0xb0 [mdc] [ 6124.430779] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6124.433854] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6124.437271] after_reply+0xe2c/0x1780 [ptlrpc] [ 6124.440602] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6124.443802] ? __next_timer_interrupt+0x160/0x160 [ 6124.446348] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6124.449164] ? do_wait_intr+0xf0/0xf0 [ 6124.451059] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6124.454193] kthread+0x1d1/0x200 [ 6124.455840] ? set_kthread_struct+0x70/0x70 [ 6124.458284] ret_from_fork+0x1f/0x30 [ 6140.053009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6140.056743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6140.057494] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6140.078321] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6140.078336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6140.078339] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30f/0x6d0 [ 6140.078355] Code: c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 <0f> 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 [ 6140.078359] RSP: 0018:ffffc38541c37500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6140.078365] RAX: 000000000000490f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6140.078367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6140.078369] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6140.078371] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6140.078383] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6140.078386] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6140.078389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6140.078391] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6140.078399] Call Trace: [ 6140.078405] [ 6140.078411] ? show_regs.cold.9+0x22/0x2f [ 6140.078420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6140.078427] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6140.078431] ? watchdog+0x40/0x40 [ 6140.078436] ? hrtimer_interrupt+0x128/0x2c0 [ 6140.078440] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6140.078450] ? apic_timer_interrupt+0xf/0x20 [ 6140.078455] [ 6140.078458] ? __pv_queued_spin_lock_slowpath+0x30f/0x6d0 [ 6140.078477] do_raw_spin_lock+0x177/0x190 [ 6140.078482] _raw_spin_lock+0x21/0x30 [ 6140.078486] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6140.078739] ? do_raw_spin_unlock+0x75/0x190 [ 6140.078746] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6140.078938] ? do_wait_intr+0xf0/0xf0 [ 6140.078946] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6140.079142] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6140.079354] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6140.079579] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6140.079684] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6140.079721] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6140.079750] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6140.079774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6140.079854] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6140.080069] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6140.080096] lmv_intent_open+0x338/0x1150 [lmv] [ 6140.080122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6140.080205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6140.080288] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6140.080312] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6140.080415] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6140.085654] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6140.085733] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6140.085741] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6140.085745] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6140.085760] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6140.085764] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6140.085769] RAX: 0000000000002d33 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6140.085771] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6140.085774] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6140.085776] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6140.085779] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6140.085782] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6140.085785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6140.085787] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6140.085797] Call Trace: [ 6140.085802] [ 6140.085807] ? show_regs.cold.9+0x22/0x2f [ 6140.085817] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6140.085824] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6140.085829] ? watchdog+0x40/0x40 [ 6140.085833] ? hrtimer_interrupt+0x128/0x2c0 [ 6140.085838] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6140.085846] ? apic_timer_interrupt+0xf/0x20 [ 6140.085850] [ 6140.085853] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6140.085860] do_raw_spin_lock+0x177/0x190 [ 6140.085865] _raw_spin_lock+0x21/0x30 [ 6140.085871] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6140.086126] ? do_raw_spin_unlock+0x75/0x190 [ 6140.086132] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6140.086325] ? do_wait_intr+0xf0/0xf0 [ 6140.086332] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6140.093889] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6140.094027] ? do_raw_spin_unlock+0x75/0x190 [ 6140.094035] ? _raw_spin_unlock+0x12/0x30 [ 6140.094040] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6140.094065] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6140.094084] ll_atomic_open+0x36a/0xd70 [lustre] [ 6140.094169] lookup_open+0x356/0xb40 [ 6140.094178] do_last+0x313/0xfc0 [ 6140.094184] ? nd_jump_root+0xe5/0x160 [ 6140.094188] ? path_init+0x437/0x520 [ 6140.094193] path_openat+0xf7/0x500 [ 6140.094197] ? do_raw_spin_unlock+0x75/0x190 [ 6140.094202] do_filp_open+0x99/0x140 [ 6140.094208] ? getname_flags+0x6e/0x330 [ 6140.094213] ? __check_object_size+0xff/0x256 [ 6140.094219] ? do_raw_spin_unlock+0x75/0x190 [ 6140.094223] ? _raw_spin_unlock+0x12/0x30 [ 6140.094227] do_sys_openat2+0x2b4/0x410 [ 6140.094231] do_sys_open+0x73/0xa0 [ 6140.094235] __x64_sys_openat+0x24/0x30 [ 6140.094239] do_syscall_64+0xc1/0x440 [ 6140.094245] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6140.094252] RIP: 0033:0x7f1a63983332 [ 6140.094259] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6140.094264] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6140.094269] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6140.094272] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6140.094274] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6140.094276] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6140.094279] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6140.379720] mdc_close+0x53b/0xd70 [mdc] [ 6140.381251] lmv_close+0x18d/0x490 [lmv] [ 6140.382864] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6140.385480] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6140.387427] ll_md_close+0x274/0x800 [lustre] [ 6140.389706] ll_file_release+0x128/0xc50 [lustre] [ 6140.392101] __fput+0x10d/0x440 [ 6140.393449] ____fput+0x12/0x20 [ 6140.394965] task_work_run+0xc6/0x110 [ 6140.396546] exit_to_usermode_loop+0x1dd/0x1f0 [ 6140.398443] do_syscall_64+0x3d6/0x440 [ 6140.400138] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6140.402707] RIP: 0033:0x7f1a63983cd8 [ 6140.404312] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6140.414767] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6140.418296] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6140.421663] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6140.424923] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6140.427929] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6140.431296] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6144.481414] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4121s! [ 6144.484361] Showing busy workqueues and worker pools: [ 6144.486337] workqueue events: flags=0x0 [ 6144.488050] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6144.490614] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6144.493809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6144.497494] pending: sec_gc_main [ptlrpc] [ 6144.499616] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6144.502186] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6144.506585] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6144.510257] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6144.513693] workqueue events_long: flags=0x0 [ 6144.515313] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6144.517730] pending: lu_objects_flush [obdclass] [ 6144.519901] workqueue events_unbound: flags=0x2 [ 6144.521652] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6144.523808] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6144.527994] workqueue events_power_efficient: flags=0x80 [ 6144.529888] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6144.532119] pending: neigh_periodic_work, do_cache_clean [ 6144.534402] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6144.537019] pending: neigh_periodic_work, check_lifetime [ 6144.539040] workqueue mm_percpu_wq: flags=0x8 [ 6144.540500] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6144.542974] pending: vmstat_update [ 6144.544182] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6144.546549] pending: vmstat_update [ 6144.547930] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6144.550182] pending: vmstat_update [ 6144.551580] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6144.554064] pending: vmstat_update [ 6144.555491] workqueue writeback: flags=0x4a [ 6144.557013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6144.559249] pending: wb_workfn, wb_workfn [ 6144.561414] workqueue xprtiod: flags=0xa [ 6144.562771] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6144.565033] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6144.567768] workqueue ipv6_addrconf: flags=0x40008 [ 6144.569937] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6144.572645] pending: addrconf_verify_work [ 6144.574185] workqueue knbd0-recv: flags=0x1a [ 6144.575795] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6144.578475] in-flight: 647:recv_work [ 6144.580276] workqueue ptlrpc_pinger: flags=0x2 [ 6144.582066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6144.584374] pending: ptlrpc_pinger_main [ptlrpc] [ 6144.586833] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6134s workers=2 idle: 652 [ 6152.057030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6152.058524] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6152.065658] Modules linked in: [ 6152.069818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6152.069907] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6152.069915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6152.069918] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6152.069931] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6152.069935] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6152.069940] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6152.069943] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6152.069945] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6152.069948] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6152.069951] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6152.070057] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6152.070061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6152.070064] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6152.070072] Call Trace: [ 6152.070077] [ 6152.070082] ? show_regs.cold.9+0x22/0x2f [ 6152.070093] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6152.070099] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6152.070103] ? watchdog+0x40/0x40 [ 6152.070108] ? hrtimer_interrupt+0x128/0x2c0 [ 6152.070112] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6152.070119] ? apic_timer_interrupt+0xf/0x20 [ 6152.070124] [ 6152.070127] ? native_safe_halt+0x2e/0x50 [ 6152.070132] kvm_wait+0xcd/0xe0 [ 6152.070138] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6152.070198] do_raw_spin_lock+0x177/0x190 [ 6152.070203] _raw_spin_lock+0x21/0x30 [ 6152.070208] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6152.070818] obd_mod_free+0x46/0x140 [ptlrpc] [ 6152.071129] mdc_commit_open+0x91/0xb0 [mdc] [ 6152.071222] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6152.071563] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6152.071951] after_reply+0xe2c/0x1780 [ptlrpc] [ 6152.072905] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6152.073282] ? __next_timer_interrupt+0x160/0x160 [ 6152.073351] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6152.073810] ? do_wait_intr+0xf0/0xf0 [ 6152.073820] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6152.074131] kthread+0x1d1/0x200 [ 6152.074137] ? set_kthread_struct+0x70/0x70 [ 6152.074141] ret_from_fork+0x1f/0x30 [ 6152.528883] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6152.555685] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6152.562864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6152.568879] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6152.571157] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6152.584074] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6152.590134] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6152.598544] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6152.606960] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6152.612132] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6152.619462] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6152.628265] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6152.631667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6152.634387] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6152.637572] Call Trace: [ 6152.638896] [ 6152.640069] ? show_regs.cold.9+0x22/0x2f [ 6152.641904] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6152.644192] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6152.645913] ? watchdog+0x40/0x40 [ 6152.647215] ? hrtimer_interrupt+0x128/0x2c0 [ 6152.649093] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6152.651472] ? apic_timer_interrupt+0xf/0x20 [ 6152.653435] [ 6152.654223] ? native_safe_halt+0x2e/0x50 [ 6152.655532] kvm_wait+0xcd/0xe0 [ 6152.657124] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6152.659457] do_raw_spin_lock+0x177/0x190 [ 6152.660822] _raw_spin_lock+0x21/0x30 [ 6152.662396] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6152.664548] ? do_raw_spin_unlock+0x75/0x190 [ 6152.666110] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6152.668792] ? do_wait_intr+0xf0/0xf0 [ 6152.670227] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6152.672383] mdc_close+0x53b/0xd70 [mdc] [ 6152.673827] lmv_close+0x18d/0x490 [lmv] [ 6152.675419] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6152.678262] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6152.680220] ll_md_close+0x274/0x800 [lustre] [ 6152.682050] ll_file_release+0x128/0xc50 [lustre] [ 6152.683936] __fput+0x10d/0x440 [ 6152.685088] ____fput+0x12/0x20 [ 6152.686359] task_work_run+0xc6/0x110 [ 6152.687619] exit_to_usermode_loop+0x1dd/0x1f0 [ 6152.689197] do_syscall_64+0x3d6/0x440 [ 6152.690517] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6152.692235] RIP: 0033:0x7f1a63983cd8 [ 6152.693771] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6152.701395] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6152.704088] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6152.706878] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6152.709752] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6152.712198] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6152.715193] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6168.060047] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6168.066701] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6168.116859] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6168.124708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6168.132857] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6168.138858] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6168.154293] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6168.157769] RAX: 000000000000329a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6168.163627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6168.167277] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6168.170711] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6168.181598] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6168.186809] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6168.193539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6168.196835] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6168.203893] Call Trace: [ 6168.205049] [ 6168.206484] ? show_regs.cold.9+0x22/0x2f [ 6168.208227] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6168.213159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6168.215659] ? watchdog+0x40/0x40 [ 6168.218395] ? hrtimer_interrupt+0x128/0x2c0 [ 6168.220603] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6168.223200] ? apic_timer_interrupt+0xf/0x20 [ 6168.225183] [ 6168.226387] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6168.228761] do_raw_spin_lock+0x177/0x190 [ 6168.230544] _raw_spin_lock+0x21/0x30 [ 6168.232641] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6168.235548] ? do_raw_spin_unlock+0x75/0x190 [ 6168.237490] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6168.240747] ? do_wait_intr+0xf0/0xf0 [ 6168.242877] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6168.246708] mdc_close+0x53b/0xd70 [mdc] [ 6168.248683] lmv_close+0x18d/0x490 [lmv] [ 6168.250856] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6168.255551] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6168.257891] ll_md_close+0x274/0x800 [lustre] [ 6168.261414] ll_file_release+0x128/0xc50 [lustre] [ 6168.263595] __fput+0x10d/0x440 [ 6168.266652] ____fput+0x12/0x20 [ 6168.270713] task_work_run+0xc6/0x110 [ 6168.273953] exit_to_usermode_loop+0x1dd/0x1f0 [ 6168.275931] do_syscall_64+0x3d6/0x440 [ 6168.279588] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6168.281629] RIP: 0033:0x7f1a63983cd8 [ 6168.284739] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6168.303459] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6168.312833] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6168.322913] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6168.330587] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6168.335024] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6168.342658] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6172.053120] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 6172.056996] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6172.083169] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6172.091896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6172.097458] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6172.100367] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6172.120252] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6172.124605] RAX: 000000000000173d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6172.136120] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6172.144899] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6172.157110] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6172.168524] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6172.181140] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6172.198358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6172.202778] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6172.216172] Call Trace: [ 6172.221631] [ 6172.229431] ? show_regs.cold.9+0x22/0x2f [ 6172.232036] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6172.233697] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6172.243022] ? watchdog+0x40/0x40 [ 6172.248218] ? hrtimer_interrupt+0x128/0x2c0 [ 6172.253986] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6172.258969] ? apic_timer_interrupt+0xf/0x20 [ 6172.267369] [ 6172.268004] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6172.273363] do_raw_spin_lock+0x177/0x190 [ 6172.275536] _raw_spin_lock+0x21/0x30 [ 6172.276999] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6172.282049] ? do_raw_spin_unlock+0x75/0x190 [ 6172.283504] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6172.286771] ? do_wait_intr+0xf0/0xf0 [ 6172.289652] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6172.291697] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6172.293595] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6172.297757] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6172.300155] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6172.303573] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6172.305670] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6172.307676] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6172.310235] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6172.315541] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6172.318623] lmv_intent_open+0x338/0x1150 [lmv] [ 6172.323220] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6172.326437] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6172.330765] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6172.334570] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6172.336728] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6172.345031] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6172.351613] ? do_raw_spin_unlock+0x75/0x190 [ 6172.354109] ? _raw_spin_unlock+0x12/0x30 [ 6172.356816] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6172.361576] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6172.364910] ll_atomic_open+0x36a/0xd70 [lustre] [ 6172.367854] lookup_open+0x356/0xb40 [ 6172.371404] do_last+0x313/0xfc0 [ 6172.372735] ? nd_jump_root+0xe5/0x160 [ 6172.374849] ? path_init+0x437/0x520 [ 6172.376924] path_openat+0xf7/0x500 [ 6172.378569] ? do_raw_spin_unlock+0x75/0x190 [ 6172.380266] do_filp_open+0x99/0x140 [ 6172.382011] ? getname_flags+0x6e/0x330 [ 6172.383304] ? __check_object_size+0xff/0x256 [ 6172.385143] ? do_raw_spin_unlock+0x75/0x190 [ 6172.388476] ? _raw_spin_unlock+0x12/0x30 [ 6172.390148] do_sys_openat2+0x2b4/0x410 [ 6172.391530] do_sys_open+0x73/0xa0 [ 6172.394697] __x64_sys_openat+0x24/0x30 [ 6172.399470] do_syscall_64+0xc1/0x440 [ 6172.400860] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6172.405456] RIP: 0033:0x7f1a63983332 [ 6172.406800] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6172.422631] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6172.426930] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6172.433410] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6172.439267] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6172.442323] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6172.449380] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6174.688337] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4151s! [ 6174.694668] Showing busy workqueues and worker pools: [ 6174.698045] workqueue events: flags=0x0 [ 6174.700559] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6174.704875] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6174.709438] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6174.713835] pending: sec_gc_main [ptlrpc] [ 6174.716811] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6174.721399] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6174.740413] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6174.746871] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6174.760872] workqueue events_long: flags=0x0 [ 6174.764270] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6174.772214] pending: lu_objects_flush [obdclass] [ 6174.777187] workqueue events_unbound: flags=0x2 [ 6174.782948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6174.787227] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6174.807477] workqueue events_power_efficient: flags=0x80 [ 6174.817286] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6174.822150] pending: neigh_periodic_work, do_cache_clean [ 6174.825508] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6174.831991] pending: neigh_periodic_work, check_lifetime [ 6174.838338] workqueue mm_percpu_wq: flags=0x8 [ 6174.846872] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6174.852637] pending: vmstat_update [ 6174.854596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6174.862620] pending: vmstat_update [ 6174.867749] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6174.875520] pending: vmstat_update [ 6174.880344] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6174.887373] pending: vmstat_update [ 6174.893170] workqueue writeback: flags=0x4a [ 6174.902496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6174.911681] pending: wb_workfn, wb_workfn [ 6174.916092] workqueue xprtiod: flags=0xa [ 6174.920973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6174.925636] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6174.936839] workqueue ipv6_addrconf: flags=0x40008 [ 6174.938647] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6174.950781] pending: addrconf_verify_work [ 6174.955737] workqueue knbd0-recv: flags=0x1a [ 6174.959568] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6174.969219] in-flight: 647:recv_work [ 6174.972227] workqueue ptlrpc_pinger: flags=0x2 [ 6174.976822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6174.984764] pending: ptlrpc_pinger_main [ptlrpc] [ 6174.991555] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6165s workers=2 idle: 652 [ 6180.057026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6180.058159] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6180.059981] Modules linked in: [ 6180.063705] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6180.063812] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6180.063819] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6180.063822] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6180.063834] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6180.063838] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6180.063842] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6180.063844] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6180.063846] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6180.063848] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6180.063851] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6180.063854] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6180.063857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6180.063859] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6180.063867] Call Trace: [ 6180.063872] [ 6180.063877] ? show_regs.cold.9+0x22/0x2f [ 6180.063887] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6180.063893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6180.063896] ? watchdog+0x40/0x40 [ 6180.063901] ? hrtimer_interrupt+0x128/0x2c0 [ 6180.063906] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6180.063912] ? apic_timer_interrupt+0xf/0x20 [ 6180.063917] [ 6180.063919] ? native_safe_halt+0x2e/0x50 [ 6180.063923] kvm_wait+0xcd/0xe0 [ 6180.063929] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6180.063935] do_raw_spin_lock+0x177/0x190 [ 6180.063939] _raw_spin_lock+0x21/0x30 [ 6180.063943] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6180.064263] obd_mod_free+0x46/0x140 [ptlrpc] [ 6180.064470] mdc_commit_open+0x91/0xb0 [mdc] [ 6180.064507] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6180.064703] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6180.064912] after_reply+0xe2c/0x1780 [ptlrpc] [ 6180.067273] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6180.067342] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6180.067349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6180.067352] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6180.067363] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6180.067368] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6180.067372] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6180.067375] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6180.067377] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6180.067379] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6180.067381] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6180.067384] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6180.067387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6180.067390] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6180.067398] Call Trace: [ 6180.067402] [ 6180.067407] ? show_regs.cold.9+0x22/0x2f [ 6180.067416] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6180.067422] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6180.067426] ? watchdog+0x40/0x40 [ 6180.067430] ? hrtimer_interrupt+0x128/0x2c0 [ 6180.067433] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6180.067440] ? apic_timer_interrupt+0xf/0x20 [ 6180.067445] [ 6180.067448] ? native_safe_halt+0x2e/0x50 [ 6180.067452] kvm_wait+0xcd/0xe0 [ 6180.067459] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6180.067466] do_raw_spin_lock+0x177/0x190 [ 6180.067470] _raw_spin_lock+0x21/0x30 [ 6180.067474] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6180.067901] ? do_raw_spin_unlock+0x75/0x190 [ 6180.067908] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6180.068178] ? do_wait_intr+0xf0/0xf0 [ 6180.068186] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6180.068388] mdc_close+0x53b/0xd70 [mdc] [ 6180.068423] lmv_close+0x18d/0x490 [lmv] [ 6180.068451] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6180.119776] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6180.129361] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6180.129479] ll_md_close+0x274/0x800 [lustre] [ 6180.129569] ll_file_release+0x128/0xc50 [lustre] [ 6180.129648] __fput+0x10d/0x440 [ 6180.129659] ____fput+0x12/0x20 [ 6180.129663] task_work_run+0xc6/0x110 [ 6180.129671] exit_to_usermode_loop+0x1dd/0x1f0 [ 6180.129678] do_syscall_64+0x3d6/0x440 [ 6180.129683] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6180.129691] RIP: 0033:0x7f1a63983cd8 [ 6180.129738] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6180.129743] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6180.129749] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6180.129751] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6180.129753] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6180.129756] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6180.129758] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6180.514956] ? __next_timer_interrupt+0x160/0x160 [ 6180.520441] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6180.522267] ? do_wait_intr+0xf0/0xf0 [ 6180.525321] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6180.531849] kthread+0x1d1/0x200 [ 6180.534394] ? set_kthread_struct+0x70/0x70 [ 6180.536416] ret_from_fork+0x1f/0x30 [ 6196.056196] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6196.060537] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6196.077217] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6196.080946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6196.083813] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6196.085540] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6196.091612] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6196.093886] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6196.096275] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6196.098313] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6196.100510] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6196.103026] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6196.105239] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6196.107589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6196.109696] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6196.112229] Call Trace: [ 6196.113202] [ 6196.113807] ? show_regs.cold.9+0x22/0x2f [ 6196.115163] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6196.116608] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6196.118208] ? watchdog+0x40/0x40 [ 6196.119306] ? hrtimer_interrupt+0x128/0x2c0 [ 6196.120611] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6196.122222] ? apic_timer_interrupt+0xf/0x20 [ 6196.123484] [ 6196.124313] ? native_safe_halt+0x2e/0x50 [ 6196.125647] kvm_wait+0xcd/0xe0 [ 6196.126638] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6196.128373] do_raw_spin_lock+0x177/0x190 [ 6196.129635] _raw_spin_lock+0x21/0x30 [ 6196.130788] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6196.132639] ? do_raw_spin_unlock+0x75/0x190 [ 6196.133999] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6196.136092] ? do_wait_intr+0xf0/0xf0 [ 6196.137197] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6196.139085] mdc_close+0x53b/0xd70 [mdc] [ 6196.140244] lmv_close+0x18d/0x490 [lmv] [ 6196.141432] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6196.143393] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6196.145070] ll_md_close+0x274/0x800 [lustre] [ 6196.146421] ll_file_release+0x128/0xc50 [lustre] [ 6196.148078] __fput+0x10d/0x440 [ 6196.149075] ____fput+0x12/0x20 [ 6196.150118] task_work_run+0xc6/0x110 [ 6196.151242] exit_to_usermode_loop+0x1dd/0x1f0 [ 6196.152513] do_syscall_64+0x3d6/0x440 [ 6196.153630] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6196.155365] RIP: 0033:0x7f1a63983cd8 [ 6196.156551] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6196.162560] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6196.165190] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6196.167396] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6196.170112] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6196.173013] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6196.176411] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6200.053009] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 6200.058704] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6200.086937] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6200.092867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6200.098298] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6200.104152] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6200.115590] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6200.119932] RAX: 0000000000000663 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6200.123489] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6200.127939] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6200.131624] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6200.135468] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6200.138913] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6200.143192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6200.145999] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6200.149902] Call Trace: [ 6200.151008] [ 6200.151965] ? show_regs.cold.9+0x22/0x2f [ 6200.154057] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6200.156722] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6200.158902] ? watchdog+0x40/0x40 [ 6200.160712] ? hrtimer_interrupt+0x128/0x2c0 [ 6200.163148] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6200.169719] ? apic_timer_interrupt+0xf/0x20 [ 6200.174824] [ 6200.175748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6200.181436] do_raw_spin_lock+0x177/0x190 [ 6200.185748] _raw_spin_lock+0x21/0x30 [ 6200.188336] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6200.192602] ? do_raw_spin_unlock+0x75/0x190 [ 6200.196442] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6200.201599] ? do_wait_intr+0xf0/0xf0 [ 6200.204651] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6200.208020] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6200.210545] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6200.217045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6200.227842] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6200.231589] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6200.239518] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6200.252040] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6200.257638] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6200.265918] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6200.268412] lmv_intent_open+0x338/0x1150 [lmv] [ 6200.273760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6200.278160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6200.282553] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6200.286538] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6200.288504] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6200.292648] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6200.297894] ? do_raw_spin_unlock+0x75/0x190 [ 6200.301999] ? _raw_spin_unlock+0x12/0x30 [ 6200.305349] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6200.309618] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6200.315303] ll_atomic_open+0x36a/0xd70 [lustre] [ 6200.319944] lookup_open+0x356/0xb40 [ 6200.325605] do_last+0x313/0xfc0 [ 6200.326588] ? nd_jump_root+0xe5/0x160 [ 6200.328192] ? path_init+0x437/0x520 [ 6200.334764] path_openat+0xf7/0x500 [ 6200.335900] ? do_raw_spin_unlock+0x75/0x190 [ 6200.343217] do_filp_open+0x99/0x140 [ 6200.345537] ? getname_flags+0x6e/0x330 [ 6200.349039] ? __check_object_size+0xff/0x256 [ 6200.354160] ? do_raw_spin_unlock+0x75/0x190 [ 6200.357285] ? _raw_spin_unlock+0x12/0x30 [ 6200.361585] do_sys_openat2+0x2b4/0x410 [ 6200.366030] do_sys_open+0x73/0xa0 [ 6200.372533] __x64_sys_openat+0x24/0x30 [ 6200.373802] do_syscall_64+0xc1/0x440 [ 6200.377604] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6200.380725] RIP: 0033:0x7f1a63983332 [ 6200.384600] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6200.396646] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6200.405892] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6200.411051] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6200.418449] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6200.426341] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6200.435198] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6205.409043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4182s! [ 6205.412117] Showing busy workqueues and worker pools: [ 6205.414287] workqueue events: flags=0x0 [ 6205.420044] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6205.428480] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6205.440557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6205.448930] pending: sec_gc_main [ptlrpc] [ 6205.453585] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6205.457930] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6205.463098] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6205.466405] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6205.475065] workqueue events_long: flags=0x0 [ 6205.478571] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6205.486350] pending: lu_objects_flush [obdclass] [ 6205.496493] workqueue events_unbound: flags=0x2 [ 6205.499258] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6205.509468] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6205.518448] workqueue events_power_efficient: flags=0x80 [ 6205.523531] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6205.532028] pending: neigh_periodic_work, do_cache_clean [ 6205.536052] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6205.541511] pending: neigh_periodic_work, check_lifetime [ 6205.545279] workqueue mm_percpu_wq: flags=0x8 [ 6205.549798] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6205.556370] pending: vmstat_update [ 6205.560707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6205.565470] pending: vmstat_update [ 6205.567242] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6205.570352] pending: vmstat_update [ 6205.572208] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6205.574985] pending: vmstat_update [ 6205.576590] workqueue writeback: flags=0x4a [ 6205.579048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6205.582129] pending: wb_workfn, wb_workfn [ 6205.583784] workqueue xprtiod: flags=0xa [ 6205.585414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6205.588818] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6205.592199] workqueue ipv6_addrconf: flags=0x40008 [ 6205.593939] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6205.596811] pending: addrconf_verify_work [ 6205.599063] workqueue knbd0-recv: flags=0x1a [ 6205.601011] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6205.603928] in-flight: 647:recv_work [ 6205.605517] workqueue ptlrpc_pinger: flags=0x2 [ 6205.611514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6205.615672] pending: ptlrpc_pinger_main [ptlrpc] [ 6205.624483] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6195s workers=2 idle: 652 [ 6208.057774] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6208.059819] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6208.063436] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6208.063877] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6208.063885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6208.063888] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6208.063901] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6208.063906] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6208.063913] RAX: 0000000000003808 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6208.063915] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6208.063917] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6208.063919] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6208.063922] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6208.063926] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6208.063929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6208.063931] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6208.063940] Call Trace: [ 6208.063946] [ 6208.063952] ? show_regs.cold.9+0x22/0x2f [ 6208.063964] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6208.063971] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6208.063975] ? watchdog+0x40/0x40 [ 6208.063979] ? hrtimer_interrupt+0x128/0x2c0 [ 6208.063983] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6208.063990] ? apic_timer_interrupt+0xf/0x20 [ 6208.063995] [ 6208.063998] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6208.064004] do_raw_spin_lock+0x177/0x190 [ 6208.064008] _raw_spin_lock+0x21/0x30 [ 6208.064013] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6208.064295] ? do_raw_spin_unlock+0x75/0x190 [ 6208.064301] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6208.064532] ? do_wait_intr+0xf0/0xf0 [ 6208.064540] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6208.065098] mdc_close+0x53b/0xd70 [mdc] [ 6208.065139] lmv_close+0x18d/0x490 [lmv] [ 6208.065170] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6208.065282] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6208.069669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6208.069815] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6208.069823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6208.069826] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6208.069839] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6208.069845] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 6208.087666] ll_md_close+0x274/0x800 [lustre] [ 6208.095606] ORIG_RAX: ffffffffffffff13 [ 6208.095615] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6208.095618] RDX: 0000000000001f00 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6208.095620] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6208.095622] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6208.095624] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6208.095626] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6208.095629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6208.095631] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6208.095637] Call Trace: [ 6208.095643] [ 6208.095648] ? show_regs.cold.9+0x22/0x2f [ 6208.095660] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6208.095666] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6208.107509] ll_file_release+0x128/0xc50 [lustre] [ 6208.110274] ? watchdog+0x40/0x40 [ 6208.118076] __fput+0x10d/0x440 [ 6208.121698] ? hrtimer_interrupt+0x128/0x2c0 [ 6208.124478] ____fput+0x12/0x20 [ 6208.124494] task_work_run+0xc6/0x110 [ 6208.124502] exit_to_usermode_loop+0x1dd/0x1f0 [ 6208.124509] do_syscall_64+0x3d6/0x440 [ 6208.124514] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6208.124522] RIP: 0033:0x7f1a63983cd8 [ 6208.124530] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6208.124533] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6208.124538] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6208.124540] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6208.124542] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6208.124544] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6208.124546] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6208.463223] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6208.466009] ? apic_timer_interrupt+0xf/0x20 [ 6208.470619] [ 6208.471716] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6208.474995] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6208.477678] do_raw_spin_lock+0x177/0x190 [ 6208.487362] _raw_spin_lock+0x21/0x30 [ 6208.492200] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6208.496343] obd_mod_free+0x46/0x140 [ptlrpc] [ 6208.501781] mdc_commit_open+0x91/0xb0 [mdc] [ 6208.504465] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6208.507849] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6208.511970] after_reply+0xe2c/0x1780 [ptlrpc] [ 6208.516912] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6208.520639] ? __next_timer_interrupt+0x160/0x160 [ 6208.522356] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6208.531574] ? do_wait_intr+0xf0/0xf0 [ 6208.534004] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6208.540930] kthread+0x1d1/0x200 [ 6208.545315] ? set_kthread_struct+0x70/0x70 [ 6208.550234] ret_from_fork+0x1f/0x30 [ 6222.501022] rcu: INFO: rcu_sched self-detected stall on CPU [ 6222.503896] rcu: 3-...!: (2582738 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 6222.509209] (t=4200103 jiffies g=269633 q=1697) [ 6222.510755] rcu: rcu_sched kthread starved for 4199750 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 6222.515154] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6222.519254] rcu: RCU grace-period kthread stack dump: [ 6222.521362] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6222.525955] Call Trace: [ 6222.527023] __schedule+0x351/0xcb0 [ 6222.528705] schedule+0xc0/0x180 [ 6222.529976] schedule_timeout+0xb4/0x190 [ 6222.531783] ? do_raw_spin_unlock+0x75/0x190 [ 6222.533613] ? __next_timer_interrupt+0x160/0x160 [ 6222.535918] ? prepare_to_swait_event+0xf3/0x140 [ 6222.537662] rcu_gp_kthread+0x8dd/0x15b0 [ 6222.539378] ? force_qs_rnp+0x2c0/0x2c0 [ 6222.541123] kthread+0x1d1/0x200 [ 6222.542333] ? set_kthread_struct+0x70/0x70 [ 6222.544623] ret_from_fork+0x1f/0x30 [ 6222.546109] rcu: Stack dump where RCU GP kthread last ran: [ 6222.548852] Sending NMI from CPU 3 to CPUs 0: [ 6222.550993] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 6222.553152] NMI backtrace for cpu 3 [ 6222.555104] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6222.560510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6222.563755] Call Trace: [ 6222.564639] [ 6222.565303] ? dump_stack+0xbb/0x10e [ 6222.566802] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6222.569168] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6222.571097] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6222.573262] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6222.575422] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6222.577513] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6222.580514] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6222.583504] ? update_process_times+0x6d/0xc0 [ 6222.586610] ? tick_sched_handle.isra.18+0x27/0x90 [ 6222.592385] ? tick_sched_timer+0x53/0xb0 [ 6222.595151] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6222.599135] ? tick_sched_handle.isra.18+0x90/0x90 [ 6222.600894] ? hrtimer_interrupt+0x128/0x2c0 [ 6222.605399] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6222.609528] ? apic_timer_interrupt+0xf/0x20 [ 6222.612309] [ 6222.613345] ? native_safe_halt+0x2e/0x50 [ 6222.618883] ? kvm_wait+0xcd/0xe0 [ 6222.632121] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6222.640115] ? do_raw_spin_lock+0x177/0x190 [ 6222.643604] ? _raw_spin_lock+0x21/0x30 [ 6222.647924] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6222.654502] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6222.663233] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6222.666697] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6222.675340] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6222.680145] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6222.688967] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6222.697844] ? __next_timer_interrupt+0x160/0x160 [ 6222.699696] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6222.710408] ? do_wait_intr+0xf0/0xf0 [ 6222.718838] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6222.728125] ? kthread+0x1d1/0x200 [ 6222.731960] ? set_kthread_struct+0x70/0x70 [ 6222.733943] ? ret_from_fork+0x1f/0x30 [ 6224.056025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6224.058907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6224.075591] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6224.079654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6224.082819] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6224.084352] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6224.090944] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6224.093662] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6224.098367] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6224.101356] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6224.103978] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6224.106532] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6224.108997] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6224.112062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6224.114145] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6224.117009] Call Trace: [ 6224.117988] [ 6224.121745] ? show_regs.cold.9+0x22/0x2f [ 6224.125361] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6224.127538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6224.131108] ? watchdog+0x40/0x40 [ 6224.132887] ? hrtimer_interrupt+0x128/0x2c0 [ 6224.136976] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6224.140315] ? apic_timer_interrupt+0xf/0x20 [ 6224.143468] [ 6224.144831] ? native_safe_halt+0x2e/0x50 [ 6224.146306] kvm_wait+0xcd/0xe0 [ 6224.147515] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6224.149339] do_raw_spin_lock+0x177/0x190 [ 6224.150929] _raw_spin_lock+0x21/0x30 [ 6224.152292] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6224.154519] ? do_raw_spin_unlock+0x75/0x190 [ 6224.155966] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6224.157755] ? do_wait_intr+0xf0/0xf0 [ 6224.158892] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6224.160968] mdc_close+0x53b/0xd70 [mdc] [ 6224.162384] lmv_close+0x18d/0x490 [lmv] [ 6224.163967] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6224.166294] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6224.168133] ll_md_close+0x274/0x800 [lustre] [ 6224.169913] ll_file_release+0x128/0xc50 [lustre] [ 6224.171852] __fput+0x10d/0x440 [ 6224.172957] ____fput+0x12/0x20 [ 6224.174023] task_work_run+0xc6/0x110 [ 6224.175333] exit_to_usermode_loop+0x1dd/0x1f0 [ 6224.177029] do_syscall_64+0x3d6/0x440 [ 6224.178308] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6224.179898] RIP: 0033:0x7f1a63983cd8 [ 6224.181054] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6224.186352] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6224.189024] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6224.190939] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6224.196868] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6224.200309] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6224.205362] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6236.057054] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6236.060317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6236.084502] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6236.089406] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6236.092890] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6236.094942] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6236.102851] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6236.106191] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6236.109299] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6236.112346] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6236.115343] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6236.118389] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6236.121423] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6236.124955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6236.127383] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6236.129038] BUG: workqueue lockup - pool [ 6236.130803] Call Trace: [ 6236.130815] [ 6236.130820] ? show_regs.cold.9+0x22/0x2f [ 6236.130834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6236.130841] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6236.130846] ? watchdog+0x40/0x40 [ 6236.130850] ? hrtimer_interrupt+0x128/0x2c0 [ 6236.130855] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6236.130862] ? apic_timer_interrupt+0xf/0x20 [ 6236.130867] [ 6236.130870] ? native_safe_halt+0x2e/0x50 [ 6236.130875] kvm_wait+0xcd/0xe0 [ 6236.130882] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6236.130892] do_raw_spin_lock+0x177/0x190 [ 6236.130897] _raw_spin_lock+0x21/0x30 [ 6236.130901] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6236.131319] ? do_raw_spin_unlock+0x75/0x190 [ 6236.131325] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6236.131672] ? do_wait_intr+0xf0/0xf0 [ 6236.131680] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6236.131912] mdc_close+0x53b/0xd70 [mdc] [ 6236.131949] lmv_close+0x18d/0x490 [lmv] [ 6236.131981] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6236.132091] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6236.134032] cpus=0-3 flags=0x4 nice=0 stuck for 4213s! [ 6236.134048] Showing busy workqueues and worker pools: [ 6236.134050] workqueue events: flags=0x0 [ 6236.134052] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6236.134060] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6236.134406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6236.134415] pending: sec_gc_main [ptlrpc] [ 6236.134872] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6236.134882] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6236.135322] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6236.135331] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6236.135738] workqueue events_long: flags=0x0 [ 6236.135741] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6236.135749] pending: lu_objects_flush [obdclass] [ 6236.135920] workqueue events_unbound: flags=0x2 [ 6236.135923] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6236.135930] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6236.135953] workqueue events_power_efficient: flags=0x80 [ 6236.135956] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6236.135963] pending: neigh_periodic_work, do_cache_clean [ 6236.135976] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6236.135983] pending: neigh_periodic_work, check_lifetime [ 6236.135996] workqueue mm_percpu_wq: flags=0x8 [ 6236.136018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6236.136025] pending: vmstat_update [ 6236.136030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6236.136036] pending: vmstat_update [ 6236.136040] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6236.136046] pending: vmstat_update [ 6236.136050] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6236.136055] pending: vmstat_update [ 6236.136063] workqueue writeback: flags=0x4a [ 6236.136065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6236.136072] pending: wb_workfn, wb_workfn [ 6236.136088] workqueue xprtiod: flags=0xa [ 6236.137168] ll_md_close+0x274/0x800 [lustre] [ 6236.137295] ll_file_release+0x128/0xc50 [lustre] [ 6236.137535] __fput+0x10d/0x440 [ 6236.137601] ____fput+0x12/0x20 [ 6236.137606] task_work_run+0xc6/0x110 [ 6236.137614] exit_to_usermode_loop+0x1dd/0x1f0 [ 6236.137621] do_syscall_64+0x3d6/0x440 [ 6236.137625] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6236.137633] RIP: 0033:0x7f1a63983cd8 [ 6236.137640] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6236.137644] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6236.137648] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6236.137650] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6236.137653] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6236.137655] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6236.137657] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6236.464154] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6236.466676] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6236.469957] workqueue ipv6_addrconf: flags=0x40008 [ 6236.472160] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6236.474792] pending: addrconf_verify_work [ 6236.476750] workqueue knbd0-recv: flags=0x1a [ 6236.478398] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6236.481127] in-flight: 647:recv_work [ 6236.482880] workqueue ptlrpc_pinger: flags=0x2 [ 6236.484853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6236.487399] pending: ptlrpc_pinger_main [ptlrpc] [ 6236.489760] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6226s workers=2 idle: 652 [ 6248.063517] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6248.078321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6248.119424] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6248.138940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6248.148015] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6248.158013] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6248.186544] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6248.196350] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6248.204090] RDX: 0000000000003a0e RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6248.211570] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6248.216727] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6248.218685] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6248.220994] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6248.224595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6248.226460] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6248.231229] Call Trace: [ 6248.232126] [ 6248.232735] ? show_regs.cold.9+0x22/0x2f [ 6248.233841] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6248.240427] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6248.241980] ? watchdog+0x40/0x40 [ 6248.245454] ? hrtimer_interrupt+0x128/0x2c0 [ 6248.247544] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6248.249656] ? apic_timer_interrupt+0xf/0x20 [ 6248.251512] [ 6248.252100] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6248.256525] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6248.258236] do_raw_spin_lock+0x177/0x190 [ 6248.259527] _raw_spin_lock+0x21/0x30 [ 6248.260757] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6248.262297] obd_mod_free+0x46/0x140 [ptlrpc] [ 6248.263598] mdc_commit_open+0x91/0xb0 [mdc] [ 6248.264788] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6248.266321] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6248.267885] after_reply+0xe2c/0x1780 [ptlrpc] [ 6248.269225] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6248.270930] ? __next_timer_interrupt+0x160/0x160 [ 6248.272178] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6248.273355] ? do_wait_intr+0xf0/0xf0 [ 6248.274452] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6248.275834] kthread+0x1d1/0x200 [ 6248.277209] ? set_kthread_struct+0x70/0x70 [ 6248.278585] ret_from_fork+0x1f/0x30 [ 6252.056017] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6252.059193] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6252.084020] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6252.091970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6252.097434] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6252.103054] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6252.113094] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6252.118010] RAX: 000000000000502b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6252.122108] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6252.126937] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6252.131795] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6252.136518] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6252.140435] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6252.145460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6252.148765] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6252.152958] Call Trace: [ 6252.154501] [ 6252.155970] ? show_regs.cold.9+0x22/0x2f [ 6252.158171] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6252.160940] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6252.163296] ? watchdog+0x40/0x40 [ 6252.165248] ? hrtimer_interrupt+0x128/0x2c0 [ 6252.167963] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6252.170954] ? apic_timer_interrupt+0xf/0x20 [ 6252.173884] [ 6252.175377] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6252.179270] do_raw_spin_lock+0x177/0x190 [ 6252.181853] _raw_spin_lock+0x21/0x30 [ 6252.183868] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6252.186667] ? do_raw_spin_unlock+0x75/0x190 [ 6252.188730] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6252.192875] ? do_wait_intr+0xf0/0xf0 [ 6252.194602] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6252.197009] mdc_close+0x53b/0xd70 [mdc] [ 6252.199069] lmv_close+0x18d/0x490 [lmv] [ 6252.201136] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6252.204672] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6252.207135] ll_md_close+0x274/0x800 [lustre] [ 6252.209475] ll_file_release+0x128/0xc50 [lustre] [ 6252.211888] __fput+0x10d/0x440 [ 6252.213111] ____fput+0x12/0x20 [ 6252.214902] task_work_run+0xc6/0x110 [ 6252.217518] exit_to_usermode_loop+0x1dd/0x1f0 [ 6252.220000] do_syscall_64+0x3d6/0x440 [ 6252.222018] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6252.224916] RIP: 0033:0x7f1a63983cd8 [ 6252.226796] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6252.235172] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6252.238803] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6252.242272] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6252.245832] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6252.249849] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6252.253162] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6264.053009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6264.056961] Modules linked in: [ 6264.057056] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6264.057065] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [ 6264.062865] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6264.062959] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6264.074128] sb_edac rapl [ 6264.099808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6264.099818] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6264.099834] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 6264.099838] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6264.099844] RAX: 0000000000003835 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6264.099847] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6264.099849] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6264.099851] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6264.099853] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6264.099856] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6264.099859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6264.099861] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6264.099870] Call Trace: [ 6264.099875] [ 6264.099880] ? show_regs.cold.9+0x22/0x2f [ 6264.099890] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6264.099897] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6264.099902] ? watchdog+0x40/0x40 [ 6264.099907] ? hrtimer_interrupt+0x128/0x2c0 [ 6264.099911] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6264.099918] ? apic_timer_interrupt+0xf/0x20 [ 6264.099934] [ 6264.099937] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6264.099944] do_raw_spin_lock+0x177/0x190 [ 6264.099949] _raw_spin_lock+0x21/0x30 [ 6264.099954] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6264.100430] ? do_raw_spin_unlock+0x75/0x190 [ 6264.100436] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6264.100639] ? do_wait_intr+0xf0/0xf0 [ 6264.100647] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6264.100910] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6264.101355] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6264.101578] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6264.129532] pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6264.129568] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6264.129575] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6264.129578] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6264.129660] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6264.129664] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6264.129670] RAX: 0000000000004f89 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6264.129672] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6264.129675] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6264.129682] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6264.129684] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6264.129688] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6264.129691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6264.129693] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6264.129702] Call Trace: [ 6264.129707] [ 6264.129713] ? show_regs.cold.9+0x22/0x2f [ 6264.129722] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6264.129728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6264.129795] ? watchdog+0x40/0x40 [ 6264.129801] ? hrtimer_interrupt+0x128/0x2c0 [ 6264.129805] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6264.129815] ? apic_timer_interrupt+0xf/0x20 [ 6264.129820] [ 6264.129823] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6264.129830] do_raw_spin_lock+0x177/0x190 [ 6264.129835] _raw_spin_lock+0x21/0x30 [ 6264.129840] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6264.130422] ? do_raw_spin_unlock+0x75/0x190 [ 6264.130435] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6264.130665] ? do_wait_intr+0xf0/0xf0 [ 6264.130673] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6264.131139] mdc_close+0x53b/0xd70 [mdc] [ 6264.131249] lmv_close+0x18d/0x490 [lmv] [ 6264.131282] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6264.131463] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6264.131561] ll_md_close+0x274/0x800 [lustre] [ 6264.139831] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6264.139884] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6264.139915] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6264.139956] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6264.140270] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6264.140638] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6264.140669] lmv_intent_open+0x338/0x1150 [lmv] [ 6264.140698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6264.140914] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6264.141112] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6264.141141] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6264.141523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6264.141621] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6264.141706] ? do_raw_spin_unlock+0x75/0x190 [ 6264.141713] ? _raw_spin_unlock+0x12/0x30 [ 6264.141720] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6264.141809] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6264.141826] ll_atomic_open+0x36a/0xd70 [lustre] [ 6264.141907] lookup_open+0x356/0xb40 [ 6264.141915] do_last+0x313/0xfc0 [ 6264.141920] ? nd_jump_root+0xe5/0x160 [ 6264.141924] ? path_init+0x437/0x520 [ 6264.141928] path_openat+0xf7/0x500 [ 6264.141933] ? do_raw_spin_unlock+0x75/0x190 [ 6264.141938] do_filp_open+0x99/0x140 [ 6264.141944] ? getname_flags+0x6e/0x330 [ 6264.141949] ? __check_object_size+0xff/0x256 [ 6264.141956] ? do_raw_spin_unlock+0x75/0x190 [ 6264.141960] ? _raw_spin_unlock+0x12/0x30 [ 6264.142274] do_sys_openat2+0x2b4/0x410 [ 6264.142281] do_sys_open+0x73/0xa0 [ 6264.142286] __x64_sys_openat+0x24/0x30 [ 6264.142290] do_syscall_64+0xc1/0x440 [ 6264.142296] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6264.142302] RIP: 0033:0x7f1a63983332 [ 6264.142309] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6264.142313] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6264.142317] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6264.142319] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6264.142321] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6264.142323] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6264.142325] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6264.721591] ll_file_release+0x128/0xc50 [lustre] [ 6264.724454] __fput+0x10d/0x440 [ 6264.725870] ____fput+0x12/0x20 [ 6264.728767] task_work_run+0xc6/0x110 [ 6264.741473] exit_to_usermode_loop+0x1dd/0x1f0 [ 6264.749751] do_syscall_64+0x3d6/0x440 [ 6264.751179] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6264.757260] RIP: 0033:0x7f1a63983cd8 [ 6264.759892] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6264.781361] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6264.793742] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6264.801906] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6264.808847] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6264.821848] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6264.826770] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6266.849927] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4243s! [ 6266.857910] Showing busy workqueues and worker pools: [ 6266.862712] workqueue events: flags=0x0 [ 6266.867754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6266.873315] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6266.887636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6266.895214] pending: sec_gc_main [ptlrpc] [ 6266.904113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6266.917777] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6266.931578] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6266.941815] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6266.956370] workqueue events_long: flags=0x0 [ 6266.963249] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6266.965066] pending: lu_objects_flush [obdclass] [ 6266.967360] workqueue events_unbound: flags=0x2 [ 6266.969042] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6266.971521] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6266.975921] workqueue events_power_efficient: flags=0x80 [ 6266.977883] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6266.980953] pending: neigh_periodic_work, do_cache_clean [ 6266.983076] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6266.985779] pending: neigh_periodic_work, check_lifetime [ 6266.987701] workqueue mm_percpu_wq: flags=0x8 [ 6266.988972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6266.990893] pending: vmstat_update [ 6266.992299] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6266.994331] pending: vmstat_update [ 6266.995326] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6266.997744] pending: vmstat_update [ 6266.999343] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6267.001274] pending: vmstat_update [ 6267.002282] workqueue writeback: flags=0x4a [ 6267.003659] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6267.005438] pending: wb_workfn, wb_workfn [ 6267.006779] workqueue xprtiod: flags=0xa [ 6267.007982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6267.009732] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6267.012529] workqueue ipv6_addrconf: flags=0x40008 [ 6267.013699] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6267.015567] pending: addrconf_verify_work [ 6267.017101] workqueue knbd0-recv: flags=0x1a [ 6267.019206] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6267.021747] in-flight: 647:recv_work [ 6267.023413] workqueue ptlrpc_pinger: flags=0x2 [ 6267.025380] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6267.028066] pending: ptlrpc_pinger_main [ptlrpc] [ 6267.030631] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6257s workers=2 idle: 652 [ 6276.060586] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6276.070743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6276.099852] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6276.105044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6276.109345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6276.112503] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6276.119751] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6276.124120] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6276.127323] RDX: 0000000000001dac RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6276.130605] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6276.134514] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6276.138833] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6276.143373] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6276.147807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6276.150742] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6276.155754] Call Trace: [ 6276.156977] [ 6276.158249] ? show_regs.cold.9+0x22/0x2f [ 6276.160507] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6276.162908] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6276.164713] ? watchdog+0x40/0x40 [ 6276.166713] ? hrtimer_interrupt+0x128/0x2c0 [ 6276.169389] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6276.172514] ? apic_timer_interrupt+0xf/0x20 [ 6276.175812] [ 6276.176954] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6276.179147] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6276.180838] do_raw_spin_lock+0x177/0x190 [ 6276.182768] _raw_spin_lock+0x21/0x30 [ 6276.185223] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6276.187917] obd_mod_free+0x46/0x140 [ptlrpc] [ 6276.191456] mdc_commit_open+0x91/0xb0 [mdc] [ 6276.194287] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6276.197244] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6276.200258] after_reply+0xe2c/0x1780 [ptlrpc] [ 6276.203517] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6276.206746] ? __next_timer_interrupt+0x160/0x160 [ 6276.209326] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6276.211423] ? do_wait_intr+0xf0/0xf0 [ 6276.213347] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6276.217791] kthread+0x1d1/0x200 [ 6276.219198] ? set_kthread_struct+0x70/0x70 [ 6276.221469] ret_from_fork+0x1f/0x30 [ 6280.056028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6280.059590] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6280.098434] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6280.107481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6280.112706] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6280.118470] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6280.135073] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6280.146103] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6280.155388] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6280.159620] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6280.164159] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6280.167771] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6280.170998] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6280.182897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6280.187379] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6280.195426] Call Trace: [ 6280.197502] [ 6280.198499] ? show_regs.cold.9+0x22/0x2f [ 6280.201335] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6280.203586] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6280.210428] ? watchdog+0x40/0x40 [ 6280.212356] ? hrtimer_interrupt+0x128/0x2c0 [ 6280.219818] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6280.225959] ? apic_timer_interrupt+0xf/0x20 [ 6280.230464] [ 6280.231968] ? native_safe_halt+0x2e/0x50 [ 6280.234344] kvm_wait+0xcd/0xe0 [ 6280.240281] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6280.242190] do_raw_spin_lock+0x177/0x190 [ 6280.243627] _raw_spin_lock+0x21/0x30 [ 6280.249378] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6280.255594] ? do_raw_spin_unlock+0x75/0x190 [ 6280.258869] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6280.264905] ? do_wait_intr+0xf0/0xf0 [ 6280.268565] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6280.275026] mdc_close+0x53b/0xd70 [mdc] [ 6280.276771] lmv_close+0x18d/0x490 [lmv] [ 6280.279673] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6280.286771] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6280.295371] ll_md_close+0x274/0x800 [lustre] [ 6280.299403] ll_file_release+0x128/0xc50 [lustre] [ 6280.307693] __fput+0x10d/0x440 [ 6280.310949] ____fput+0x12/0x20 [ 6280.313753] task_work_run+0xc6/0x110 [ 6280.317956] exit_to_usermode_loop+0x1dd/0x1f0 [ 6280.321358] do_syscall_64+0x3d6/0x440 [ 6280.323061] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6280.326063] RIP: 0033:0x7f1a63983cd8 [ 6280.327279] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6280.339855] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6280.346723] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6280.353542] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6280.361375] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6280.367609] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6280.370718] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6292.053020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6292.056851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 6292.057148] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6292.057154] intel_rapl_common sb_edac rapl [ 6292.083410] Modules linked in: [ 6292.087740] pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6292.087817] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6292.087825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6292.087828] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6292.087850] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6292.087854] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6292.087859] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6292.087862] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6292.087864] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6292.087866] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6292.087869] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6292.087872] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6292.087876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6292.087878] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6292.087886] Call Trace: [ 6292.087968] [ 6292.087976] ? show_regs.cold.9+0x22/0x2f [ 6292.087986] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6292.087994] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6292.087999] ? watchdog+0x40/0x40 [ 6292.088004] ? hrtimer_interrupt+0x128/0x2c0 [ 6292.088008] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6292.088014] ? apic_timer_interrupt+0xf/0x20 [ 6292.088019] [ 6292.088022] ? native_safe_halt+0x2e/0x50 [ 6292.088026] kvm_wait+0xcd/0xe0 [ 6292.088033] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6292.088041] do_raw_spin_lock+0x177/0x190 [ 6292.088045] _raw_spin_lock+0x21/0x30 [ 6292.088048] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6292.088549] ? do_raw_spin_unlock+0x75/0x190 [ 6292.088556] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6292.088828] ? do_wait_intr+0xf0/0xf0 [ 6292.088836] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6292.089107] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6292.089555] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6292.096704] lustre(O) [ 6292.098246] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6292.098537] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6292.098576] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6292.098608] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6292.098637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6292.098723] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6292.117292] osp(O) ofd(O) [ 6292.123842] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6292.123912] lmv_intent_open+0x338/0x1150 [lmv] [ 6292.123946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6292.124209] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6292.124292] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6292.124321] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6292.124635] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6292.124735] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6292.124838] ? do_raw_spin_unlock+0x75/0x190 [ 6292.124847] ? _raw_spin_unlock+0x12/0x30 [ 6292.124853] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6292.124884] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6292.124980] ll_atomic_open+0x36a/0xd70 [lustre] [ 6292.137768] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6292.137845] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6292.137853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6292.137856] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6292.137874] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6292.137878] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6292.137882] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6292.137893] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6292.137907] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6292.137909] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6292.137912] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6292.137915] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6292.137918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6292.137921] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6292.137929] Call Trace: [ 6292.137934] [ 6292.137940] ? show_regs.cold.9+0x22/0x2f [ 6292.137950] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6292.137958] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6292.137962] ? watchdog+0x40/0x40 [ 6292.137967] ? hrtimer_interrupt+0x128/0x2c0 [ 6292.140322] lookup_open+0x356/0xb40 [ 6292.140345] do_last+0x313/0xfc0 [ 6292.140419] ? nd_jump_root+0xe5/0x160 [ 6292.140426] ? path_init+0x437/0x520 [ 6292.140431] path_openat+0xf7/0x500 [ 6292.140437] ? do_raw_spin_unlock+0x75/0x190 [ 6292.140444] do_filp_open+0x99/0x140 [ 6292.140452] ? getname_flags+0x6e/0x330 [ 6292.140457] ? __check_object_size+0xff/0x256 [ 6292.140464] ? do_raw_spin_unlock+0x75/0x190 [ 6292.140468] ? _raw_spin_unlock+0x12/0x30 [ 6292.140479] do_sys_openat2+0x2b4/0x410 [ 6292.140484] do_sys_open+0x73/0xa0 [ 6292.140489] __x64_sys_openat+0x24/0x30 [ 6292.140493] do_syscall_64+0xc1/0x440 [ 6292.140500] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6292.140507] RIP: 0033:0x7f1a63983332 [ 6292.140835] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6292.140845] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6292.140851] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6292.140854] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6292.140857] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6292.140859] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6292.140862] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6292.462291] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6292.465875] ? apic_timer_interrupt+0xf/0x20 [ 6292.468669] [ 6292.473809] ? native_safe_halt+0x2e/0x50 [ 6292.478282] kvm_wait+0xcd/0xe0 [ 6292.480655] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6292.483793] do_raw_spin_lock+0x177/0x190 [ 6292.485848] _raw_spin_lock+0x21/0x30 [ 6292.488756] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6292.492662] ? do_raw_spin_unlock+0x75/0x190 [ 6292.494551] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6292.497560] ? do_wait_intr+0xf0/0xf0 [ 6292.499537] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6292.503319] mdc_close+0x53b/0xd70 [mdc] [ 6292.505432] lmv_close+0x18d/0x490 [lmv] [ 6292.506752] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6292.510549] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6292.513134] ll_md_close+0x274/0x800 [lustre] [ 6292.515807] ll_file_release+0x128/0xc50 [lustre] [ 6292.518970] __fput+0x10d/0x440 [ 6292.523720] ____fput+0x12/0x20 [ 6292.526668] task_work_run+0xc6/0x110 [ 6292.530576] exit_to_usermode_loop+0x1dd/0x1f0 [ 6292.533787] do_syscall_64+0x3d6/0x440 [ 6292.538765] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6292.544909] RIP: 0033:0x7f1a63983cd8 [ 6292.546698] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6292.563686] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6292.570972] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6292.577118] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6292.584083] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6292.589231] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6292.593228] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6297.057763] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4274s! [ 6297.065787] Showing busy workqueues and worker pools: [ 6297.071370] workqueue events: flags=0x0 [ 6297.074454] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6297.081574] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6297.090641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6297.099463] pending: sec_gc_main [ptlrpc] [ 6297.105398] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6297.114273] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6297.127760] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6297.134643] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6297.140875] workqueue events_long: flags=0x0 [ 6297.143529] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6297.151059] pending: lu_objects_flush [obdclass] [ 6297.156617] workqueue events_unbound: flags=0x2 [ 6297.157951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6297.166769] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6297.172120] workqueue events_power_efficient: flags=0x80 [ 6297.177530] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6297.179982] pending: neigh_periodic_work, do_cache_clean [ 6297.191010] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6297.198654] pending: neigh_periodic_work, check_lifetime [ 6297.207415] workqueue mm_percpu_wq: flags=0x8 [ 6297.213392] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6297.219481] pending: vmstat_update [ 6297.221436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6297.224753] pending: vmstat_update [ 6297.227356] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6297.233415] pending: vmstat_update [ 6297.235078] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6297.241372] pending: vmstat_update [ 6297.245822] workqueue writeback: flags=0x4a [ 6297.250726] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6297.256328] pending: wb_workfn, wb_workfn [ 6297.259937] workqueue xprtiod: flags=0xa [ 6297.264514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6297.271027] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6297.278898] workqueue ipv6_addrconf: flags=0x40008 [ 6297.282293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6297.290641] pending: addrconf_verify_work [ 6297.292528] workqueue knbd0-recv: flags=0x1a [ 6297.294208] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6297.299922] in-flight: 647:recv_work [ 6297.301562] workqueue ptlrpc_pinger: flags=0x2 [ 6297.305566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6297.310669] pending: ptlrpc_pinger_main [ptlrpc] [ 6297.315593] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6287s workers=2 idle: 652 [ 6304.058022] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6304.060689] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6304.080989] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6304.085705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6304.089514] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6304.091513] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6304.098875] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6304.101888] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6304.104734] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6304.107903] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6304.110766] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6304.113879] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6304.116948] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6304.120368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6304.122857] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6304.125720] Call Trace: [ 6304.126756] [ 6304.127656] ? show_regs.cold.9+0x22/0x2f [ 6304.129426] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6304.131216] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6304.132995] ? watchdog+0x40/0x40 [ 6304.134336] ? hrtimer_interrupt+0x128/0x2c0 [ 6304.136053] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6304.138112] ? apic_timer_interrupt+0xf/0x20 [ 6304.139847] [ 6304.140597] ? native_safe_halt+0x2e/0x50 [ 6304.142364] kvm_wait+0xcd/0xe0 [ 6304.143624] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6304.145759] do_raw_spin_lock+0x177/0x190 [ 6304.147556] _raw_spin_lock+0x21/0x30 [ 6304.149157] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6304.151584] obd_mod_free+0x46/0x140 [ptlrpc] [ 6304.153936] mdc_commit_open+0x91/0xb0 [mdc] [ 6304.155928] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6304.158252] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6304.160638] after_reply+0xe2c/0x1780 [ptlrpc] [ 6304.162748] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6304.165140] ? __next_timer_interrupt+0x160/0x160 [ 6304.167111] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6304.169114] ? do_wait_intr+0xf0/0xf0 [ 6304.170646] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6304.173045] kthread+0x1d1/0x200 [ 6304.174399] ? set_kthread_struct+0x70/0x70 [ 6304.176039] ret_from_fork+0x1f/0x30 [ 6308.056072] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6308.060660] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6308.099539] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6308.103387] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6308.110704] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6308.115379] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6308.128002] RSP: 0018:ffffc38542073af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6308.132890] RAX: 0000000000005365 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6308.145699] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6308.151287] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6308.157150] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6308.161003] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6308.164739] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6308.172485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6308.175103] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6308.179344] Call Trace: [ 6308.182859] [ 6308.183585] ? show_regs.cold.9+0x22/0x2f [ 6308.184951] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6308.189581] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6308.191377] ? watchdog+0x40/0x40 [ 6308.197415] ? hrtimer_interrupt+0x128/0x2c0 [ 6308.200144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6308.202140] ? apic_timer_interrupt+0xf/0x20 [ 6308.208378] [ 6308.211101] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6308.214086] do_raw_spin_lock+0x177/0x190 [ 6308.216680] _raw_spin_lock+0x21/0x30 [ 6308.222296] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6308.226710] ? do_raw_spin_unlock+0x75/0x190 [ 6308.228477] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6308.230809] ? do_wait_intr+0xf0/0xf0 [ 6308.232067] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6308.234161] mdc_close+0x53b/0xd70 [mdc] [ 6308.235571] lmv_close+0x18d/0x490 [lmv] [ 6308.237002] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6308.238962] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6308.240613] ll_md_close+0x274/0x800 [lustre] [ 6308.242231] ll_file_release+0x128/0xc50 [lustre] [ 6308.244054] __fput+0x10d/0x440 [ 6308.245176] ____fput+0x12/0x20 [ 6308.246250] task_work_run+0xc6/0x110 [ 6308.247575] exit_to_usermode_loop+0x1dd/0x1f0 [ 6308.249199] do_syscall_64+0x3d6/0x440 [ 6308.250580] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6308.252366] RIP: 0033:0x7f1a63983cd8 [ 6308.253618] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6308.260640] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6308.263569] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6308.266318] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6308.268521] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6308.271110] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6308.273689] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6320.057029] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 6320.060033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6320.127850] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6320.153947] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6320.161765] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6320.165698] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6320.193340] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6320.207705] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6320.213681] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6320.224365] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6320.228518] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6320.235503] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6320.241925] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6320.254609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6320.263206] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6320.268313] Call Trace: [ 6320.269276] [ 6320.270059] ? show_regs.cold.9+0x22/0x2f [ 6320.285206] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6320.287145] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6320.301595] ? watchdog+0x40/0x40 [ 6320.302852] ? hrtimer_interrupt+0x128/0x2c0 [ 6320.316887] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6320.324012] ? apic_timer_interrupt+0xf/0x20 [ 6320.327346] [ 6320.328658] ? native_safe_halt+0x2e/0x50 [ 6320.334611] kvm_wait+0xcd/0xe0 [ 6320.338461] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6320.347837] do_raw_spin_lock+0x177/0x190 [ 6320.358156] _raw_spin_lock+0x21/0x30 [ 6320.360667] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6320.376431] ? do_raw_spin_unlock+0x75/0x190 [ 6320.377898] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6320.388376] ? do_wait_intr+0xf0/0xf0 [ 6320.394950] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6320.406558] mdc_close+0x53b/0xd70 [mdc] [ 6320.409365] lmv_close+0x18d/0x490 [lmv] [ 6320.417602] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6320.423452] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6320.427588] ll_md_close+0x274/0x800 [lustre] [ 6320.434436] ll_file_release+0x128/0xc50 [lustre] [ 6320.436109] __fput+0x10d/0x440 [ 6320.437009] ____fput+0x12/0x20 [ 6320.446260] task_work_run+0xc6/0x110 [ 6320.452681] exit_to_usermode_loop+0x1dd/0x1f0 [ 6320.455545] do_syscall_64+0x3d6/0x440 [ 6320.464670] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6320.468580] RIP: 0033:0x7f1a63983cd8 [ 6320.471279] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6320.497429] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6320.503407] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6320.516687] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6320.522309] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6320.528372] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6320.534808] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6324.053024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6324.057554] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6324.083076] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6324.089388] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6324.096330] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6324.099543] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6324.110415] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6324.115407] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6324.119905] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6324.124416] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6324.131489] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6324.139134] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6324.144887] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6324.163715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6324.168083] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6324.172317] Call Trace: [ 6324.174169] [ 6324.175372] ? show_regs.cold.9+0x22/0x2f [ 6324.179248] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6324.189837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6324.193869] ? watchdog+0x40/0x40 [ 6324.198248] ? hrtimer_interrupt+0x128/0x2c0 [ 6324.203450] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6324.212398] ? apic_timer_interrupt+0xf/0x20 [ 6324.220431] [ 6324.221364] ? native_safe_halt+0x2e/0x50 [ 6324.223373] kvm_wait+0xcd/0xe0 [ 6324.228088] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6324.231747] do_raw_spin_lock+0x177/0x190 [ 6324.237137] _raw_spin_lock+0x21/0x30 [ 6324.239908] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6324.244313] ? do_raw_spin_unlock+0x75/0x190 [ 6324.256363] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6324.262429] ? do_wait_intr+0xf0/0xf0 [ 6324.265093] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6324.272380] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6324.282215] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6324.292400] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6324.299783] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6324.302781] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6324.309244] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6324.313732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6324.318938] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6324.325515] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6324.328401] lmv_intent_open+0x338/0x1150 [lmv] [ 6324.331946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6324.340968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6324.344862] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6324.349032] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6324.352911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6324.356849] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6324.360379] ? do_raw_spin_unlock+0x75/0x190 [ 6324.364039] ? _raw_spin_unlock+0x12/0x30 [ 6324.366288] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6324.370451] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6324.376808] ll_atomic_open+0x36a/0xd70 [lustre] [ 6324.382272] lookup_open+0x356/0xb40 [ 6324.387439] do_last+0x313/0xfc0 [ 6324.392615] ? nd_jump_root+0xe5/0x160 [ 6324.395230] ? path_init+0x437/0x520 [ 6324.397224] path_openat+0xf7/0x500 [ 6324.402229] ? do_raw_spin_unlock+0x75/0x190 [ 6324.407674] do_filp_open+0x99/0x140 [ 6324.409488] ? getname_flags+0x6e/0x330 [ 6324.412344] ? __check_object_size+0xff/0x256 [ 6324.417030] ? do_raw_spin_unlock+0x75/0x190 [ 6324.419466] ? _raw_spin_unlock+0x12/0x30 [ 6324.421933] do_sys_openat2+0x2b4/0x410 [ 6324.423796] do_sys_open+0x73/0xa0 [ 6324.425883] __x64_sys_openat+0x24/0x30 [ 6324.428266] do_syscall_64+0xc1/0x440 [ 6324.430230] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6324.433225] RIP: 0033:0x7f1a63983332 [ 6324.442711] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6324.452269] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6324.457061] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6324.461021] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6324.464749] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6324.468224] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6324.471814] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6327.781857] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4304s! [ 6327.784534] Showing busy workqueues and worker pools: [ 6327.799383] workqueue events: flags=0x0 [ 6327.804190] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6327.807636] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6327.819443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6327.825676] pending: sec_gc_main [ptlrpc] [ 6327.829863] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6327.835017] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6327.842219] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6327.853732] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6327.862600] workqueue events_long: flags=0x0 [ 6327.868508] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6327.871390] pending: lu_objects_flush [obdclass] [ 6327.878790] workqueue events_unbound: flags=0x2 [ 6327.880447] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6327.888158] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6327.897969] workqueue events_power_efficient: flags=0x80 [ 6327.906240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6327.915099] pending: neigh_periodic_work, do_cache_clean [ 6327.920200] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6327.929488] pending: neigh_periodic_work, check_lifetime [ 6327.935627] workqueue mm_percpu_wq: flags=0x8 [ 6327.939336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6327.945309] pending: vmstat_update [ 6327.946542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6327.965265] pending: vmstat_update [ 6327.967910] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6327.976532] pending: vmstat_update [ 6327.977864] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6327.990031] pending: vmstat_update [ 6327.991412] workqueue writeback: flags=0x4a [ 6327.996163] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6328.001126] pending: wb_workfn, wb_workfn [ 6328.005110] workqueue xprtiod: flags=0xa [ 6328.011948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6328.014042] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6328.031408] workqueue ipv6_addrconf: flags=0x40008 [ 6328.035525] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6328.045268] pending: addrconf_verify_work [ 6328.050769] workqueue knbd0-recv: flags=0x1a [ 6328.052140] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6328.060202] in-flight: 647:recv_work [ 6328.065115] workqueue ptlrpc_pinger: flags=0x2 [ 6328.069492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6328.075570] pending: ptlrpc_pinger_main [ptlrpc] [ 6328.083264] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6318s workers=2 idle: 652 [ 6332.061468] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6332.067205] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6332.114823] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6332.124009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6332.135256] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6332.141241] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6332.157404] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6332.164246] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6332.170088] RDX: 0000000000001887 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6332.178328] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6332.183764] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6332.192912] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6332.198295] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6332.202464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6332.208341] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6332.213224] Call Trace: [ 6332.214429] [ 6332.216396] ? show_regs.cold.9+0x22/0x2f [ 6332.218360] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6332.221949] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6332.226291] ? watchdog+0x40/0x40 [ 6332.228443] ? hrtimer_interrupt+0x128/0x2c0 [ 6332.230341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6332.235238] ? apic_timer_interrupt+0xf/0x20 [ 6332.237288] [ 6332.238596] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6332.240980] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6332.243589] do_raw_spin_lock+0x177/0x190 [ 6332.245372] _raw_spin_lock+0x21/0x30 [ 6332.248447] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6332.250368] obd_mod_free+0x46/0x140 [ptlrpc] [ 6332.253173] mdc_commit_open+0x91/0xb0 [mdc] [ 6332.257888] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6332.260454] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6332.265766] after_reply+0xe2c/0x1780 [ptlrpc] [ 6332.268305] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6332.271293] ? __next_timer_interrupt+0x160/0x160 [ 6332.275012] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6332.276732] ? do_wait_intr+0xf0/0xf0 [ 6332.279304] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6332.287146] kthread+0x1d1/0x200 [ 6332.293160] ? set_kthread_struct+0x70/0x70 [ 6332.296178] ret_from_fork+0x1f/0x30 [ 6336.056026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6336.059250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6336.077263] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6336.086354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6336.102032] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6336.106798] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6336.134614] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6336.143640] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6336.151403] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6336.156676] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6336.160413] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6336.166909] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6336.174697] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6336.180967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6336.183877] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6336.187885] Call Trace: [ 6336.188806] [ 6336.190726] ? show_regs.cold.9+0x22/0x2f [ 6336.192765] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6336.196777] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6336.203390] ? watchdog+0x40/0x40 [ 6336.206165] ? hrtimer_interrupt+0x128/0x2c0 [ 6336.209856] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6336.215670] ? apic_timer_interrupt+0xf/0x20 [ 6336.217449] [ 6336.218731] ? native_safe_halt+0x2e/0x50 [ 6336.224454] kvm_wait+0xcd/0xe0 [ 6336.231610] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6336.236277] do_raw_spin_lock+0x177/0x190 [ 6336.238449] _raw_spin_lock+0x21/0x30 [ 6336.245268] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6336.249716] ? do_raw_spin_unlock+0x75/0x190 [ 6336.253240] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6336.261623] ? do_wait_intr+0xf0/0xf0 [ 6336.265594] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6336.269673] mdc_close+0x53b/0xd70 [mdc] [ 6336.271909] lmv_close+0x18d/0x490 [lmv] [ 6336.275690] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6336.281580] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6336.283982] ll_md_close+0x274/0x800 [lustre] [ 6336.287974] ll_file_release+0x128/0xc50 [lustre] [ 6336.292761] __fput+0x10d/0x440 [ 6336.296292] ____fput+0x12/0x20 [ 6336.299355] task_work_run+0xc6/0x110 [ 6336.300581] exit_to_usermode_loop+0x1dd/0x1f0 [ 6336.301644] do_syscall_64+0x3d6/0x440 [ 6336.314290] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6336.317902] RIP: 0033:0x7f1a63983cd8 [ 6336.321917] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6336.335018] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6336.354297] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6336.363015] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6336.372436] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6336.377960] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6336.389621] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6348.057096] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 6348.064678] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6348.098279] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6348.103076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6348.108829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6348.112729] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6348.120149] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6348.123865] RAX: 000000000000630e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6348.128144] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6348.132204] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6348.140306] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6348.143737] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6348.146858] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6348.150856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6348.156963] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6348.163984] Call Trace: [ 6348.165491] [ 6348.166668] ? show_regs.cold.9+0x22/0x2f [ 6348.172508] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6348.178501] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6348.182225] ? watchdog+0x40/0x40 [ 6348.186293] ? hrtimer_interrupt+0x128/0x2c0 [ 6348.193105] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6348.199226] ? apic_timer_interrupt+0xf/0x20 [ 6348.203544] [ 6348.206940] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6348.209501] do_raw_spin_lock+0x177/0x190 [ 6348.211685] _raw_spin_lock+0x21/0x30 [ 6348.214166] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6348.220688] ? do_raw_spin_unlock+0x75/0x190 [ 6348.226137] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6348.229393] ? do_wait_intr+0xf0/0xf0 [ 6348.236238] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6348.241916] mdc_close+0x53b/0xd70 [mdc] [ 6348.246905] lmv_close+0x18d/0x490 [lmv] [ 6348.253052] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6348.259260] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6348.262449] ll_md_close+0x274/0x800 [lustre] [ 6348.265627] ll_file_release+0x128/0xc50 [lustre] [ 6348.268316] __fput+0x10d/0x440 [ 6348.270121] ____fput+0x12/0x20 [ 6348.271383] task_work_run+0xc6/0x110 [ 6348.275527] exit_to_usermode_loop+0x1dd/0x1f0 [ 6348.276919] do_syscall_64+0x3d6/0x440 [ 6348.280282] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6348.282647] RIP: 0033:0x7f1a63983cd8 [ 6348.283649] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6348.288863] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6348.290945] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6348.292953] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6348.295005] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6348.297076] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6348.298981] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6356.053046] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6356.059819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6356.121289] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6356.139665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6356.159698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6356.161897] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6356.201606] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6356.205960] RAX: 0000000000007a91 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6356.211076] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6356.216841] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6356.222048] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6356.225040] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6356.228433] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6356.231046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6356.232988] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6356.235620] Call Trace: [ 6356.236192] [ 6356.237020] ? show_regs.cold.9+0x22/0x2f [ 6356.240695] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6356.242959] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6356.245186] ? watchdog+0x40/0x40 [ 6356.250108] ? hrtimer_interrupt+0x128/0x2c0 [ 6356.251961] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6356.254128] ? apic_timer_interrupt+0xf/0x20 [ 6356.257716] [ 6356.258724] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6356.264211] do_raw_spin_lock+0x177/0x190 [ 6356.267135] _raw_spin_lock+0x21/0x30 [ 6356.270799] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6356.275154] ? do_raw_spin_unlock+0x75/0x190 [ 6356.279666] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6356.282921] ? do_wait_intr+0xf0/0xf0 [ 6356.285157] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6356.294159] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6356.298067] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6356.304043] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6356.311489] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6356.315092] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6356.318835] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6356.320180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6356.322586] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6356.325023] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6356.327238] lmv_intent_open+0x338/0x1150 [lmv] [ 6356.329650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6356.332681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6356.335103] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6356.337264] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6356.339789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6356.342378] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6356.347017] ? do_raw_spin_unlock+0x75/0x190 [ 6356.349124] ? _raw_spin_unlock+0x12/0x30 [ 6356.350975] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6356.353773] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6356.360761] ll_atomic_open+0x36a/0xd70 [lustre] [ 6356.363707] lookup_open+0x356/0xb40 [ 6356.369019] do_last+0x313/0xfc0 [ 6356.372833] ? nd_jump_root+0xe5/0x160 [ 6356.373914] ? path_init+0x437/0x520 [ 6356.375028] path_openat+0xf7/0x500 [ 6356.376674] ? do_raw_spin_unlock+0x75/0x190 [ 6356.378665] do_filp_open+0x99/0x140 [ 6356.380235] ? getname_flags+0x6e/0x330 [ 6356.381968] ? __check_object_size+0xff/0x256 [ 6356.383954] ? do_raw_spin_unlock+0x75/0x190 [ 6356.385754] ? _raw_spin_unlock+0x12/0x30 [ 6356.389971] do_sys_openat2+0x2b4/0x410 [ 6356.391154] do_sys_open+0x73/0xa0 [ 6356.394270] __x64_sys_openat+0x24/0x30 [ 6356.396323] do_syscall_64+0xc1/0x440 [ 6356.401038] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6356.403403] RIP: 0033:0x7f1a63983332 [ 6356.405209] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6356.422037] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6356.427654] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6356.433946] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6356.437520] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6356.440788] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6356.447395] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6358.497687] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4335s! [ 6358.500703] Showing busy workqueues and worker pools: [ 6358.502636] workqueue events: flags=0x0 [ 6358.504624] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6358.507436] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6358.513901] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6358.516101] pending: sec_gc_main [ptlrpc] [ 6358.522698] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6358.528813] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6358.533617] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6358.536241] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6358.539986] workqueue events_long: flags=0x0 [ 6358.541669] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6358.547508] pending: lu_objects_flush [obdclass] [ 6358.553063] workqueue events_unbound: flags=0x2 [ 6358.557155] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6358.560190] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6358.565238] workqueue events_power_efficient: flags=0x80 [ 6358.568497] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6358.572735] pending: neigh_periodic_work, do_cache_clean [ 6358.576329] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6358.579620] pending: neigh_periodic_work, check_lifetime [ 6358.588961] workqueue mm_percpu_wq: flags=0x8 [ 6358.596027] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6358.605663] pending: vmstat_update [ 6358.606887] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6358.609710] pending: vmstat_update [ 6358.611466] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6358.615791] pending: vmstat_update [ 6358.617764] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6358.620196] pending: vmstat_update [ 6358.623051] workqueue writeback: flags=0x4a [ 6358.629546] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6358.638145] pending: wb_workfn, wb_workfn [ 6358.644356] workqueue xprtiod: flags=0xa [ 6358.647738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6358.657225] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6358.660657] workqueue ipv6_addrconf: flags=0x40008 [ 6358.664515] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6358.672813] pending: addrconf_verify_work [ 6358.677546] workqueue knbd0-recv: flags=0x1a [ 6358.679089] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6358.691157] in-flight: 647:recv_work [ 6358.696227] workqueue ptlrpc_pinger: flags=0x2 [ 6358.701576] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6358.707625] pending: ptlrpc_pinger_main [ptlrpc] [ 6358.713856] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6348s workers=2 idle: 652 [ 6360.058284] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6360.061850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6360.083655] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6360.089098] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6360.093065] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6360.095607] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6360.104918] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6360.109460] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6360.112744] RDX: 0000000000003b60 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6360.115678] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6360.118946] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6360.122114] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6360.125404] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6360.128818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6360.131327] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6360.134347] Call Trace: [ 6360.135372] [ 6360.136092] ? show_regs.cold.9+0x22/0x2f [ 6360.137926] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6360.142744] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6360.144553] ? watchdog+0x40/0x40 [ 6360.146225] ? hrtimer_interrupt+0x128/0x2c0 [ 6360.148685] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6360.150640] ? apic_timer_interrupt+0xf/0x20 [ 6360.152822] [ 6360.153826] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6360.156468] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6360.158958] do_raw_spin_lock+0x177/0x190 [ 6360.160668] _raw_spin_lock+0x21/0x30 [ 6360.162525] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6360.164579] obd_mod_free+0x46/0x140 [ptlrpc] [ 6360.166821] mdc_commit_open+0x91/0xb0 [mdc] [ 6360.168671] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6360.170783] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6360.173430] after_reply+0xe2c/0x1780 [ptlrpc] [ 6360.175692] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6360.178104] ? __next_timer_interrupt+0x160/0x160 [ 6360.180345] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6360.182565] ? do_wait_intr+0xf0/0xf0 [ 6360.184087] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6360.186514] kthread+0x1d1/0x200 [ 6360.188033] ? set_kthread_struct+0x70/0x70 [ 6360.190064] ret_from_fork+0x1f/0x30 [ 6364.056837] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6364.067888] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6364.133228] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6364.148109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6364.157161] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6364.166896] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6364.187969] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6364.192693] RAX: 000000000000109b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6364.198145] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6364.203975] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6364.207528] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6364.210790] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6364.215607] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6364.221669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6364.225675] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6364.233472] Call Trace: [ 6364.236489] [ 6364.239838] ? show_regs.cold.9+0x22/0x2f [ 6364.244535] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6364.250051] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6364.256035] ? watchdog+0x40/0x40 [ 6364.259165] ? hrtimer_interrupt+0x128/0x2c0 [ 6364.266781] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6364.275335] ? apic_timer_interrupt+0xf/0x20 [ 6364.280735] [ 6364.285712] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6364.290287] do_raw_spin_lock+0x177/0x190 [ 6364.293026] _raw_spin_lock+0x21/0x30 [ 6364.296483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6364.300144] ? do_raw_spin_unlock+0x75/0x190 [ 6364.302698] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6364.309636] ? do_wait_intr+0xf0/0xf0 [ 6364.315259] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6364.323491] mdc_close+0x53b/0xd70 [mdc] [ 6364.327761] lmv_close+0x18d/0x490 [lmv] [ 6364.332616] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6364.342034] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6364.349556] ll_md_close+0x274/0x800 [lustre] [ 6364.357737] ll_file_release+0x128/0xc50 [lustre] [ 6364.359391] __fput+0x10d/0x440 [ 6364.362245] ____fput+0x12/0x20 [ 6364.363624] task_work_run+0xc6/0x110 [ 6364.364901] exit_to_usermode_loop+0x1dd/0x1f0 [ 6364.366718] do_syscall_64+0x3d6/0x440 [ 6364.368300] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6364.370509] RIP: 0033:0x7f1a63983cd8 [ 6364.371962] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6364.391929] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6364.407658] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6364.409847] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6364.425415] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6364.432719] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6364.441683] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6376.060556] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 6376.063036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6376.106341] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6376.115489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6376.120601] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6376.123934] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6376.141605] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6376.153718] RAX: 0000000000004a85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6376.156889] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6376.162897] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6376.169074] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6376.173511] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6376.183828] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6376.185967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6376.188885] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6376.195059] Call Trace: [ 6376.195744] [ 6376.196587] ? show_regs.cold.9+0x22/0x2f [ 6376.198144] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6376.201632] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6376.205344] ? watchdog+0x40/0x40 [ 6376.208191] ? hrtimer_interrupt+0x128/0x2c0 [ 6376.212543] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6376.217454] ? apic_timer_interrupt+0xf/0x20 [ 6376.222247] [ 6376.223024] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6376.225982] do_raw_spin_lock+0x177/0x190 [ 6376.229803] _raw_spin_lock+0x21/0x30 [ 6376.231736] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6376.236334] ? do_raw_spin_unlock+0x75/0x190 [ 6376.238997] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6376.242998] ? do_wait_intr+0xf0/0xf0 [ 6376.244323] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6376.248625] mdc_close+0x53b/0xd70 [mdc] [ 6376.252031] lmv_close+0x18d/0x490 [lmv] [ 6376.253474] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6376.258509] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6376.261304] ll_md_close+0x274/0x800 [lustre] [ 6376.263821] ll_file_release+0x128/0xc50 [lustre] [ 6376.268702] __fput+0x10d/0x440 [ 6376.272670] ____fput+0x12/0x20 [ 6376.277608] task_work_run+0xc6/0x110 [ 6376.278928] exit_to_usermode_loop+0x1dd/0x1f0 [ 6376.281236] do_syscall_64+0x3d6/0x440 [ 6376.282683] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6376.285638] RIP: 0033:0x7f1a63983cd8 [ 6376.287531] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6376.298604] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6376.301975] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6376.305198] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6376.308371] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6376.311539] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6376.314823] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6384.053025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6384.056094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6384.121899] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6384.151887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6384.160556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6384.164718] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6384.180243] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6384.189772] RAX: 0000000000007e89 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6384.196431] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6384.200947] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6384.204345] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6384.207509] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6384.211199] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6384.216180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6384.219719] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6384.223617] Call Trace: [ 6384.224975] [ 6384.227628] ? show_regs.cold.9+0x22/0x2f [ 6384.230159] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6384.232129] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6384.233794] ? watchdog+0x40/0x40 [ 6384.235193] ? hrtimer_interrupt+0x128/0x2c0 [ 6384.237661] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6384.241203] ? apic_timer_interrupt+0xf/0x20 [ 6384.243524] [ 6384.244578] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6384.250720] do_raw_spin_lock+0x177/0x190 [ 6384.252766] _raw_spin_lock+0x21/0x30 [ 6384.254263] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6384.256285] ? do_raw_spin_unlock+0x75/0x190 [ 6384.257710] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6384.259843] ? do_wait_intr+0xf0/0xf0 [ 6384.263319] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6384.266528] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6384.269395] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6384.271435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6384.273524] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6384.275656] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6384.277270] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6384.278955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6384.281227] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6384.283491] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6384.285387] lmv_intent_open+0x338/0x1150 [lmv] [ 6384.287132] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6384.289178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6384.291134] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6384.292792] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6384.294911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6384.297149] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6384.298957] ? do_raw_spin_unlock+0x75/0x190 [ 6384.300655] ? _raw_spin_unlock+0x12/0x30 [ 6384.302239] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6384.304108] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6384.306066] ll_atomic_open+0x36a/0xd70 [lustre] [ 6384.307821] lookup_open+0x356/0xb40 [ 6384.309206] do_last+0x313/0xfc0 [ 6384.310673] ? nd_jump_root+0xe5/0x160 [ 6384.312200] ? path_init+0x437/0x520 [ 6384.313695] path_openat+0xf7/0x500 [ 6384.315091] ? do_raw_spin_unlock+0x75/0x190 [ 6384.316738] do_filp_open+0x99/0x140 [ 6384.318136] ? getname_flags+0x6e/0x330 [ 6384.319475] ? __check_object_size+0xff/0x256 [ 6384.320888] ? do_raw_spin_unlock+0x75/0x190 [ 6384.322846] ? _raw_spin_unlock+0x12/0x30 [ 6384.324357] do_sys_openat2+0x2b4/0x410 [ 6384.325832] do_sys_open+0x73/0xa0 [ 6384.326937] __x64_sys_openat+0x24/0x30 [ 6384.328157] do_syscall_64+0xc1/0x440 [ 6384.329495] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6384.331422] RIP: 0033:0x7f1a63983332 [ 6384.332545] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6384.339514] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6384.341683] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6384.344382] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6384.346972] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6384.349436] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6384.351940] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6388.058008] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6388.062261] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6388.081347] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6388.086218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6388.089900] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6388.091883] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6388.099138] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6388.101665] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6388.103820] RDX: 0000000000004a1d RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6388.106426] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6388.108412] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6388.110758] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6388.112938] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6388.115707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6388.117321] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6388.120054] Call Trace: [ 6388.121003] [ 6388.121914] ? show_regs.cold.9+0x22/0x2f [ 6388.123573] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6388.125374] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6388.127250] ? watchdog+0x40/0x40 [ 6388.128536] ? hrtimer_interrupt+0x128/0x2c0 [ 6388.130346] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6388.132219] ? apic_timer_interrupt+0xf/0x20 [ 6388.133649] [ 6388.134311] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6388.135801] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6388.137503] do_raw_spin_lock+0x177/0x190 [ 6388.138830] _raw_spin_lock+0x21/0x30 [ 6388.140199] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6388.141938] obd_mod_free+0x46/0x140 [ptlrpc] [ 6388.143526] mdc_commit_open+0x91/0xb0 [mdc] [ 6388.145404] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6388.147346] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6388.149218] after_reply+0xe2c/0x1780 [ptlrpc] [ 6388.150753] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6388.152636] ? __next_timer_interrupt+0x160/0x160 [ 6388.154543] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6388.156243] ? do_wait_intr+0xf0/0xf0 [ 6388.157531] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6388.159370] kthread+0x1d1/0x200 [ 6388.160332] ? set_kthread_struct+0x70/0x70 [ 6388.161661] ret_from_fork+0x1f/0x30 [ 6389.216293] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4366s! [ 6389.219153] Showing busy workqueues and worker pools: [ 6389.221204] workqueue events: flags=0x0 [ 6389.222395] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6389.224971] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6389.227811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6389.230122] pending: sec_gc_main [ptlrpc] [ 6389.231834] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6389.235275] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6389.238626] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6389.240918] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6389.244362] workqueue events_long: flags=0x0 [ 6389.245795] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6389.248090] pending: lu_objects_flush [obdclass] [ 6389.250035] workqueue events_unbound: flags=0x2 [ 6389.251470] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6389.253716] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6389.257550] workqueue events_power_efficient: flags=0x80 [ 6389.259479] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6389.261843] pending: neigh_periodic_work, do_cache_clean [ 6389.263939] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6389.266260] pending: neigh_periodic_work, check_lifetime [ 6389.268242] workqueue mm_percpu_wq: flags=0x8 [ 6389.270128] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6389.272187] pending: vmstat_update [ 6389.273429] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6389.275813] pending: vmstat_update [ 6389.277613] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6389.280102] pending: vmstat_update [ 6389.281546] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6389.283892] pending: vmstat_update [ 6389.285374] workqueue writeback: flags=0x4a [ 6389.286819] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6389.288709] pending: wb_workfn, wb_workfn [ 6389.290211] workqueue xprtiod: flags=0xa [ 6389.291626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6389.293513] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6389.296214] workqueue ipv6_addrconf: flags=0x40008 [ 6389.297691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6389.299858] pending: addrconf_verify_work [ 6389.301346] workqueue knbd0-recv: flags=0x1a [ 6389.302755] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6389.305071] in-flight: 647:recv_work [ 6389.306328] workqueue ptlrpc_pinger: flags=0x2 [ 6389.308015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6389.309948] pending: ptlrpc_pinger_main [ptlrpc] [ 6389.312127] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6379s workers=2 idle: 652 [ 6392.056109] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 6392.059419] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6392.097826] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6392.102203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6392.108511] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6392.112537] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6392.130720] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6392.136601] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6392.146241] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6392.152342] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6392.155963] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6392.159128] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6392.162240] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6392.165640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6392.168040] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6392.175507] Call Trace: [ 6392.176323] [ 6392.179702] ? show_regs.cold.9+0x22/0x2f [ 6392.182131] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6392.185420] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6392.190793] ? watchdog+0x40/0x40 [ 6392.192381] ? hrtimer_interrupt+0x128/0x2c0 [ 6392.201764] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6392.207263] ? apic_timer_interrupt+0xf/0x20 [ 6392.209322] [ 6392.212327] ? native_safe_halt+0x2e/0x50 [ 6392.218039] kvm_wait+0xcd/0xe0 [ 6392.223244] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6392.226427] do_raw_spin_lock+0x177/0x190 [ 6392.231381] _raw_spin_lock+0x21/0x30 [ 6392.233474] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6392.235860] ? do_raw_spin_unlock+0x75/0x190 [ 6392.245909] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6392.249116] ? do_wait_intr+0xf0/0xf0 [ 6392.252862] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6392.258511] mdc_close+0x53b/0xd70 [mdc] [ 6392.263734] lmv_close+0x18d/0x490 [lmv] [ 6392.269347] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6392.273914] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6392.279163] ll_md_close+0x274/0x800 [lustre] [ 6392.284367] ll_file_release+0x128/0xc50 [lustre] [ 6392.287899] __fput+0x10d/0x440 [ 6392.289860] ____fput+0x12/0x20 [ 6392.291223] task_work_run+0xc6/0x110 [ 6392.298279] exit_to_usermode_loop+0x1dd/0x1f0 [ 6392.300727] do_syscall_64+0x3d6/0x440 [ 6392.302812] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6392.306661] RIP: 0033:0x7f1a63983cd8 [ 6392.310213] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6392.325116] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6392.329670] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6392.342085] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6392.344442] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6392.347403] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6392.350280] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6402.508524] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 6402.521531] rcu: 3-...!: (2699439 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 6402.538333] (detected by 1, t=4380102 jiffies, g=269633, q=1750) [ 6402.542346] Sending NMI from CPU 1 to CPUs 3: [ 6402.552499] NMI backtrace for cpu 3 [ 6402.552507] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6402.552509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6402.552511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6402.552512] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6402.552514] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 6402.552517] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6402.552518] RDX: 0000000000004f50 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6402.552519] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6402.552524] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6402.552526] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6402.552527] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6402.552529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6402.552530] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6402.552532] Call Trace: [ 6402.552533] [ 6402.552534] ? show_regs.cold.9+0x22/0x2f [ 6402.552535] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6402.552536] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6402.552537] ? nmi_handle+0xbd/0x210 [ 6402.552538] ? default_do_nmi+0x4b/0x1b0 [ 6402.552539] ? do_nmi+0x280/0x380 [ 6402.552540] ? end_repeat_nmi+0x16/0x1f [ 6402.552541] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6402.552542] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6402.552544] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6402.552545] [ 6402.552546] do_raw_spin_lock+0x177/0x190 [ 6402.552547] _raw_spin_lock+0x21/0x30 [ 6402.552551] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6402.552552] obd_mod_free+0x46/0x140 [ptlrpc] [ 6402.552553] mdc_commit_open+0x91/0xb0 [mdc] [ 6402.552554] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6402.552555] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6402.552556] after_reply+0xe2c/0x1780 [ptlrpc] [ 6402.552557] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6402.552559] ? __next_timer_interrupt+0x160/0x160 [ 6402.552560] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6402.552561] ? do_wait_intr+0xf0/0xf0 [ 6402.552562] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6402.552563] kthread+0x1d1/0x200 [ 6402.552564] ? set_kthread_struct+0x70/0x70 [ 6402.552565] ret_from_fork+0x1f/0x30 [ 6402.556322] rcu: rcu_sched kthread starved for 4379748 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 6402.865028] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6402.878256] rcu: RCU grace-period kthread stack dump: [ 6402.894240] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6402.906129] Call Trace: [ 6402.913888] __schedule+0x351/0xcb0 [ 6402.915216] schedule+0xc0/0x180 [ 6402.916456] schedule_timeout+0xb4/0x190 [ 6402.921299] ? do_raw_spin_unlock+0x75/0x190 [ 6402.924313] ? __next_timer_interrupt+0x160/0x160 [ 6402.928215] ? prepare_to_swait_event+0xf3/0x140 [ 6402.931369] rcu_gp_kthread+0x8dd/0x15b0 [ 6402.934215] ? force_qs_rnp+0x2c0/0x2c0 [ 6402.937883] kthread+0x1d1/0x200 [ 6402.939720] ? set_kthread_struct+0x70/0x70 [ 6402.944249] ret_from_fork+0x1f/0x30 [ 6402.946241] rcu: Stack dump where RCU GP kthread last ran: [ 6402.952929] Sending NMI from CPU 1 to CPUs 2: [ 6402.958730] NMI backtrace for cpu 2 [ 6402.958736] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6402.958739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6402.958741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6402.958742] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6402.958745] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 [ 6402.958747] RAX: 0000000000007520 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6402.958749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6402.958750] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6402.958752] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6402.958753] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6402.958755] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6402.958756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6402.958758] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6402.958759] Call Trace: [ 6402.958760] [ 6402.958760] ? show_regs.cold.9+0x22/0x2f [ 6402.958761] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6402.958762] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6402.958763] ? nmi_handle+0xbd/0x210 [ 6402.958764] ? default_do_nmi+0x4b/0x1b0 [ 6402.958765] ? do_nmi+0x280/0x380 [ 6402.958766] ? end_repeat_nmi+0x16/0x1f [ 6402.958768] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6402.958769] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6402.958770] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6402.958771] [ 6402.958772] do_raw_spin_lock+0x177/0x190 [ 6402.958773] _raw_spin_lock+0x21/0x30 [ 6402.958774] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6402.958775] ? do_raw_spin_unlock+0x75/0x190 [ 6402.958776] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6402.958777] ? do_wait_intr+0xf0/0xf0 [ 6402.958778] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6402.958780] mdc_close+0x53b/0xd70 [mdc] [ 6402.958781] lmv_close+0x18d/0x490 [lmv] [ 6402.958782] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6402.958783] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6402.958784] ll_md_close+0x274/0x800 [lustre] [ 6402.958786] ll_file_release+0x128/0xc50 [lustre] [ 6402.958787] __fput+0x10d/0x440 [ 6402.958788] ____fput+0x12/0x20 [ 6402.958789] task_work_run+0xc6/0x110 [ 6402.958790] exit_to_usermode_loop+0x1dd/0x1f0 [ 6402.958791] do_syscall_64+0x3d6/0x440 [ 6402.958792] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6402.958793] RIP: 0033:0x7f1a63983cd8 [ 6402.958795] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6402.958797] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6402.958799] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6402.958801] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6402.958802] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6402.958803] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6402.958805] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6416.053032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6416.057151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6416.083872] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6416.089158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6416.093116] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6416.095910] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6416.104853] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6416.108492] RAX: 000000000000463b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6416.111819] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6416.115258] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6416.118728] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6416.121815] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6416.125224] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6416.129224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6416.132183] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6416.135543] Call Trace: [ 6416.136669] [ 6416.137512] ? show_regs.cold.9+0x22/0x2f [ 6416.139458] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6416.141754] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6416.143810] ? watchdog+0x40/0x40 [ 6416.145302] ? hrtimer_interrupt+0x128/0x2c0 [ 6416.147281] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6416.150016] ? apic_timer_interrupt+0xf/0x20 [ 6416.152080] [ 6416.153054] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6416.155573] do_raw_spin_lock+0x177/0x190 [ 6416.157438] _raw_spin_lock+0x21/0x30 [ 6416.159164] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6416.161676] ? do_raw_spin_unlock+0x75/0x190 [ 6416.163419] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6416.165986] ? do_wait_intr+0xf0/0xf0 [ 6416.167489] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6416.170020] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6416.171882] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6416.174232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6416.176414] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6416.178786] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6416.180937] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6416.182885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6416.185281] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6416.188119] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6416.190644] lmv_intent_open+0x338/0x1150 [lmv] [ 6416.192802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6416.195108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6416.197466] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6416.199453] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6416.201646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6416.204146] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6416.206254] ? do_raw_spin_unlock+0x75/0x190 [ 6416.208103] ? _raw_spin_unlock+0x12/0x30 [ 6416.210143] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6416.212490] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6416.214869] ll_atomic_open+0x36a/0xd70 [lustre] [ 6416.217201] lookup_open+0x356/0xb40 [ 6416.218803] do_last+0x313/0xfc0 [ 6416.220246] ? nd_jump_root+0xe5/0x160 [ 6416.221961] ? path_init+0x437/0x520 [ 6416.223669] path_openat+0xf7/0x500 [ 6416.225112] ? do_raw_spin_unlock+0x75/0x190 [ 6416.227110] do_filp_open+0x99/0x140 [ 6416.228894] ? getname_flags+0x6e/0x330 [ 6416.230649] ? __check_object_size+0xff/0x256 [ 6416.232574] ? do_raw_spin_unlock+0x75/0x190 [ 6416.234482] ? _raw_spin_unlock+0x12/0x30 [ 6416.236240] do_sys_openat2+0x2b4/0x410 [ 6416.237933] do_sys_open+0x73/0xa0 [ 6416.239426] __x64_sys_openat+0x24/0x30 [ 6416.241062] do_syscall_64+0xc1/0x440 [ 6416.242626] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6416.244821] RIP: 0033:0x7f1a63983332 [ 6416.246448] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6416.254962] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6416.258229] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6416.261302] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6416.264311] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6416.272046] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6416.281241] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6419.430928] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4396s! [ 6419.438452] Showing busy workqueues and worker pools: [ 6419.451379] workqueue events: flags=0x0 [ 6419.456146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6419.466954] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6419.475772] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6419.479510] pending: sec_gc_main [ptlrpc] [ 6419.486272] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6419.497313] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6419.510242] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6419.515961] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6419.523494] workqueue events_long: flags=0x0 [ 6419.530431] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6419.535633] pending: lu_objects_flush [obdclass] [ 6419.542530] workqueue events_unbound: flags=0x2 [ 6419.549042] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6419.560727] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6419.573646] workqueue events_power_efficient: flags=0x80 [ 6419.581780] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6419.588003] pending: neigh_periodic_work, do_cache_clean [ 6419.592506] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6419.599225] pending: neigh_periodic_work, check_lifetime [ 6419.602364] workqueue mm_percpu_wq: flags=0x8 [ 6419.607356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6419.614793] pending: vmstat_update [ 6419.619896] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6419.628288] pending: vmstat_update [ 6419.632129] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6419.639818] pending: vmstat_update [ 6419.646273] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6419.649628] pending: vmstat_update [ 6419.653850] workqueue writeback: flags=0x4a [ 6419.659248] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6419.669353] pending: wb_workfn, wb_workfn [ 6419.672675] workqueue xprtiod: flags=0xa [ 6419.676668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6419.683768] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6419.697067] workqueue ipv6_addrconf: flags=0x40008 [ 6419.703391] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6419.710344] pending: addrconf_verify_work [ 6419.715458] workqueue knbd0-recv: flags=0x1a [ 6419.720468] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6419.729652] in-flight: 647:recv_work [ 6419.734879] workqueue ptlrpc_pinger: flags=0x2 [ 6419.740939] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6419.747513] pending: ptlrpc_pinger_main [ptlrpc] [ 6419.753517] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6409s workers=2 idle: 652 [ 6428.056035] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6428.057016] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6428.059408] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6428.059754] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6428.059762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6428.059765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6428.059779] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6428.059784] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6428.059788] RAX: 0000000000000582 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6428.059791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6428.059793] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6428.059796] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6428.059798] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6428.059802] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6428.059805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6428.059808] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6428.059817] Call Trace: [ 6428.059831] [ 6428.059837] ? show_regs.cold.9+0x22/0x2f [ 6428.059846] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6428.059852] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6428.059856] ? watchdog+0x40/0x40 [ 6428.059861] ? hrtimer_interrupt+0x128/0x2c0 [ 6428.059865] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6428.059874] ? apic_timer_interrupt+0xf/0x20 [ 6428.059878] [ 6428.059881] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6428.059887] do_raw_spin_lock+0x177/0x190 [ 6428.059892] _raw_spin_lock+0x21/0x30 [ 6428.059897] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6428.060209] ? do_raw_spin_unlock+0x75/0x190 [ 6428.060215] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6428.060434] ? do_wait_intr+0xf0/0xf0 [ 6428.060443] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6428.067205] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6428.067218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6428.067303] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6428.067308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6428.067311] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6428.067325] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6428.067328] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6428.067333] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6428.067335] RDX: 00000000000079f0 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6428.067338] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6428.067340] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6428.067342] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6428.067346] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6428.067349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6428.067351] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6428.067360] Call Trace: [ 6428.067365] [ 6428.067371] ? show_regs.cold.9+0x22/0x2f [ 6428.067381] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6428.067387] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6428.067391] ? watchdog+0x40/0x40 [ 6428.067395] ? hrtimer_interrupt+0x128/0x2c0 [ 6428.067399] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6428.067406] ? apic_timer_interrupt+0xf/0x20 [ 6428.067411] [ 6428.067414] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6428.067419] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6428.067425] do_raw_spin_lock+0x177/0x190 [ 6428.067430] _raw_spin_lock+0x21/0x30 [ 6428.067435] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6428.068848] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul [ 6428.067731] obd_mod_free+0x46/0x140 [ptlrpc] [ 6428.070949] mdc_commit_open+0x91/0xb0 [mdc] [ 6428.070989] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6428.071200] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6428.071396] after_reply+0xe2c/0x1780 [ptlrpc] [ 6428.071600] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6428.074472] ? __next_timer_interrupt+0x160/0x160 [ 6428.074486] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6428.074726] ? do_wait_intr+0xf0/0xf0 [ 6428.074735] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6428.075004] kthread+0x1d1/0x200 [ 6428.075010] ? set_kthread_struct+0x70/0x70 [ 6428.075014] ret_from_fork+0x1f/0x30 [ 6428.089033] mdc_close+0x53b/0xd70 [mdc] [ 6428.089085] lmv_close+0x18d/0x490 [lmv] [ 6428.112150] crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6428.112176] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6428.112183] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6428.112185] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6428.112198] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6428.112204] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6428.112208] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6428.112211] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6428.112213] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6428.112216] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6428.112219] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6428.116340] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6428.116465] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6428.116564] ll_md_close+0x274/0x800 [lustre] [ 6428.117288] ll_file_release+0x128/0xc50 [lustre] [ 6428.117388] __fput+0x10d/0x440 [ 6428.117398] ____fput+0x12/0x20 [ 6428.117410] task_work_run+0xc6/0x110 [ 6428.117419] exit_to_usermode_loop+0x1dd/0x1f0 [ 6428.117427] do_syscall_64+0x3d6/0x440 [ 6428.117434] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6428.117448] RIP: 0033:0x7f1a63983cd8 [ 6428.117455] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6428.117459] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6428.117465] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6428.117468] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6428.117470] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6428.117473] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6428.117479] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6428.766493] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6428.770772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6428.773231] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6428.775487] Call Trace: [ 6428.776368] [ 6428.778055] ? show_regs.cold.9+0x22/0x2f [ 6428.780965] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6428.783458] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6428.786068] ? watchdog+0x40/0x40 [ 6428.787930] ? hrtimer_interrupt+0x128/0x2c0 [ 6428.790176] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6428.792556] ? apic_timer_interrupt+0xf/0x20 [ 6428.794769] [ 6428.796076] ? native_safe_halt+0x2e/0x50 [ 6428.798050] kvm_wait+0xcd/0xe0 [ 6428.799724] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6428.802358] do_raw_spin_lock+0x177/0x190 [ 6428.804685] _raw_spin_lock+0x21/0x30 [ 6428.806664] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6428.809698] ? do_raw_spin_unlock+0x75/0x190 [ 6428.812021] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6428.815811] ? do_wait_intr+0xf0/0xf0 [ 6428.818096] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6428.820288] mdc_close+0x53b/0xd70 [mdc] [ 6428.822445] lmv_close+0x18d/0x490 [lmv] [ 6428.824260] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6428.826940] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6428.829391] ll_md_close+0x274/0x800 [lustre] [ 6428.831346] ll_file_release+0x128/0xc50 [lustre] [ 6428.834349] __fput+0x10d/0x440 [ 6428.835750] ____fput+0x12/0x20 [ 6428.837124] task_work_run+0xc6/0x110 [ 6428.838517] exit_to_usermode_loop+0x1dd/0x1f0 [ 6428.840382] do_syscall_64+0x3d6/0x440 [ 6428.842010] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6428.844934] RIP: 0033:0x7f1a63983cd8 [ 6428.846410] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6428.855403] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6428.858710] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6428.862176] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6428.865136] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6428.868009] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6428.871173] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6444.054825] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6444.063163] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6444.107132] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6444.113660] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6444.118723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6444.122121] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6444.132047] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6444.135758] RAX: 000000000000378c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6444.139731] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6444.143519] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6444.146668] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6444.150198] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6444.153214] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6444.157361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6444.160105] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6444.165311] Call Trace: [ 6444.166833] [ 6444.167976] ? show_regs.cold.9+0x22/0x2f [ 6444.170889] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6444.173695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6444.176195] ? watchdog+0x40/0x40 [ 6444.178176] ? hrtimer_interrupt+0x128/0x2c0 [ 6444.180653] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6444.183267] ? apic_timer_interrupt+0xf/0x20 [ 6444.185572] [ 6444.186982] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6444.189659] do_raw_spin_lock+0x177/0x190 [ 6444.191892] _raw_spin_lock+0x21/0x30 [ 6444.193704] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6444.197206] ? do_raw_spin_unlock+0x75/0x190 [ 6444.199304] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6444.202161] ? do_wait_intr+0xf0/0xf0 [ 6444.204189] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6444.207411] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6444.210970] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6444.214962] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6444.217999] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6444.221629] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6444.224475] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6444.227556] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6444.230410] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6444.235088] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6444.239465] lmv_intent_open+0x338/0x1150 [lmv] [ 6444.242177] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6444.244924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6444.248109] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6444.250105] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6444.252519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6444.255269] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6444.258551] ? do_raw_spin_unlock+0x75/0x190 [ 6444.261629] ? _raw_spin_unlock+0x12/0x30 [ 6444.263937] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6444.267229] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6444.270315] ll_atomic_open+0x36a/0xd70 [lustre] [ 6444.273284] lookup_open+0x356/0xb40 [ 6444.275952] do_last+0x313/0xfc0 [ 6444.277872] ? nd_jump_root+0xe5/0x160 [ 6444.279998] ? path_init+0x437/0x520 [ 6444.285722] path_openat+0xf7/0x500 [ 6444.288755] ? do_raw_spin_unlock+0x75/0x190 [ 6444.294266] do_filp_open+0x99/0x140 [ 6444.301134] ? getname_flags+0x6e/0x330 [ 6444.302914] ? __check_object_size+0xff/0x256 [ 6444.308970] ? do_raw_spin_unlock+0x75/0x190 [ 6444.312103] ? _raw_spin_unlock+0x12/0x30 [ 6444.315207] do_sys_openat2+0x2b4/0x410 [ 6444.317780] do_sys_open+0x73/0xa0 [ 6444.319987] __x64_sys_openat+0x24/0x30 [ 6444.322406] do_syscall_64+0xc1/0x440 [ 6444.324742] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6444.327264] RIP: 0033:0x7f1a63983332 [ 6444.330064] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6444.340273] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6444.344131] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6444.348071] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6444.351227] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6444.355062] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6444.358740] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6450.146017] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4427s! [ 6450.159032] Showing busy workqueues and worker pools: [ 6450.166256] workqueue events: flags=0x0 [ 6450.171638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6450.178071] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6450.191629] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6450.202887] pending: sec_gc_main [ptlrpc] [ 6450.209709] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6450.212754] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6450.219019] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6450.225293] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6450.232929] workqueue events_long: flags=0x0 [ 6450.237787] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6450.243271] pending: lu_objects_flush [obdclass] [ 6450.246156] workqueue events_unbound: flags=0x2 [ 6450.251242] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6450.261988] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6450.273301] workqueue events_power_efficient: flags=0x80 [ 6450.275286] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6450.287929] pending: neigh_periodic_work, do_cache_clean [ 6450.299032] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6450.308016] pending: neigh_periodic_work, check_lifetime [ 6450.318850] workqueue mm_percpu_wq: flags=0x8 [ 6450.326716] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6450.336750] pending: vmstat_update [ 6450.343944] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6450.354750] pending: vmstat_update [ 6450.360234] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6450.366858] pending: vmstat_update [ 6450.375876] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6450.389084] pending: vmstat_update [ 6450.394853] workqueue writeback: flags=0x4a [ 6450.402748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6450.411588] pending: wb_workfn, wb_workfn [ 6450.423260] workqueue xprtiod: flags=0xa [ 6450.425885] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6450.437886] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6450.449206] workqueue ipv6_addrconf: flags=0x40008 [ 6450.454894] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6450.467445] pending: addrconf_verify_work [ 6450.475661] workqueue knbd0-recv: flags=0x1a [ 6450.481050] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6450.489157] in-flight: 647:recv_work [ 6450.496409] workqueue ptlrpc_pinger: flags=0x2 [ 6450.498717] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6450.506757] pending: ptlrpc_pinger_main [ptlrpc] [ 6450.515085] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6440s workers=2 idle: 652 [ 6456.056067] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6456.057012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6456.058202] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6456.058210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6456.058318] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6456.058323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6456.058326] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6456.058433] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6456.058438] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6456.058443] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6456.058446] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6456.058448] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6456.058451] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6456.058454] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6456.058457] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6456.058461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6456.058463] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6456.058471] Call Trace: [ 6456.058477] [ 6456.058482] ? show_regs.cold.9+0x22/0x2f [ 6456.058492] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6456.058498] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6456.058502] ? watchdog+0x40/0x40 [ 6456.058507] ? hrtimer_interrupt+0x128/0x2c0 [ 6456.058512] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6456.058519] ? apic_timer_interrupt+0xf/0x20 [ 6456.058524] [ 6456.058528] ? native_safe_halt+0x2e/0x50 [ 6456.058533] kvm_wait+0xcd/0xe0 [ 6456.058540] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6456.058547] do_raw_spin_lock+0x177/0x190 [ 6456.058552] _raw_spin_lock+0x21/0x30 [ 6456.058557] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6456.058884] obd_mod_free+0x46/0x140 [ptlrpc] [ 6456.059012] Modules linked in: [ 6456.059088] mdc_commit_open+0x91/0xb0 [mdc] [ 6456.059126] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6456.060853] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6456.060909] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6456.060640] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6456.060915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6456.060918] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6456.060926] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6456.060929] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6456.060932] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6456.060934] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6456.060935] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6456.060937] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6456.060938] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6456.060941] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6456.060943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6456.060945] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6456.060952] Call Trace: [ 6456.060956] [ 6456.060959] ? show_regs.cold.9+0x22/0x2f [ 6456.060970] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6456.060975] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6456.060979] ? watchdog+0x40/0x40 [ 6456.060984] ? hrtimer_interrupt+0x128/0x2c0 [ 6456.060988] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6456.060995] ? apic_timer_interrupt+0xf/0x20 [ 6456.060998] [ 6456.061000] ? native_safe_halt+0x2e/0x50 [ 6456.061003] kvm_wait+0xcd/0xe0 [ 6456.061009] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6456.061014] do_raw_spin_lock+0x177/0x190 [ 6456.061019] _raw_spin_lock+0x21/0x30 [ 6456.061025] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6456.061271] ? do_raw_spin_unlock+0x75/0x190 [ 6456.061274] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6456.061027] after_reply+0xe2c/0x1780 [ptlrpc] [ 6456.062164] ? do_wait_intr+0xf0/0xf0 [ 6456.062170] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6456.062352] mdc_close+0x53b/0xd70 [mdc] [ 6456.062389] lmv_close+0x18d/0x490 [lmv] [ 6456.062423] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6456.062749] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6456.062801] ll_md_close+0x274/0x800 [lustre] [ 6456.063156] ll_file_release+0x128/0xc50 [lustre] [ 6456.063243] __fput+0x10d/0x440 [ 6456.063250] ____fput+0x12/0x20 [ 6456.063253] task_work_run+0xc6/0x110 [ 6456.063258] exit_to_usermode_loop+0x1dd/0x1f0 [ 6456.063263] do_syscall_64+0x3d6/0x440 [ 6456.063266] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6456.063271] RIP: 0033:0x7f1a63983cd8 [ 6456.063278] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6456.063281] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6456.063284] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6456.063286] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6456.063287] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6456.063289] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6456.063291] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6456.062966] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6456.071534] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6456.071613] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6456.071621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6456.071625] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6456.071638] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6456.071642] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6456.071647] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6456.071650] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6456.071652] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6456.071654] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6456.071657] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6456.071660] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6456.071663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6456.071665] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6456.071674] Call Trace: [ 6456.071679] [ 6456.071685] ? show_regs.cold.9+0x22/0x2f [ 6456.071696] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6456.071703] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6456.071707] ? watchdog+0x40/0x40 [ 6456.071712] ? hrtimer_interrupt+0x128/0x2c0 [ 6456.071716] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6456.071722] ? apic_timer_interrupt+0xf/0x20 [ 6456.071727] [ 6456.071730] ? native_safe_halt+0x2e/0x50 [ 6456.071735] kvm_wait+0xcd/0xe0 [ 6456.071742] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6456.071749] do_raw_spin_lock+0x177/0x190 [ 6456.071753] _raw_spin_lock+0x21/0x30 [ 6456.071757] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6456.121487] ? __next_timer_interrupt+0x160/0x160 [ 6456.121511] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6456.122871] ? do_wait_intr+0xf0/0xf0 [ 6456.122882] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6456.130297] ? do_raw_spin_unlock+0x75/0x190 [ 6456.141337] kthread+0x1d1/0x200 [ 6456.143294] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6456.163287] ? set_kthread_struct+0x70/0x70 [ 6456.163303] ret_from_fork+0x1f/0x30 [ 6456.170066] ? do_wait_intr+0xf0/0xf0 [ 6456.983575] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6456.991774] mdc_close+0x53b/0xd70 [mdc] [ 6456.993178] lmv_close+0x18d/0x490 [lmv] [ 6456.996507] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6457.000477] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6457.002786] ll_md_close+0x274/0x800 [lustre] [ 6457.010960] ll_file_release+0x128/0xc50 [lustre] [ 6457.012487] __fput+0x10d/0x440 [ 6457.013503] ____fput+0x12/0x20 [ 6457.018135] task_work_run+0xc6/0x110 [ 6457.019516] exit_to_usermode_loop+0x1dd/0x1f0 [ 6457.023579] do_syscall_64+0x3d6/0x440 [ 6457.026037] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6457.028166] RIP: 0033:0x7f1a63983cd8 [ 6457.029281] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6457.044658] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6457.052740] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6457.059236] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6457.066172] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6457.073026] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6457.075197] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6476.062102] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 6476.068184] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6476.091794] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6476.097290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6476.101701] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6476.104675] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6476.116820] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6476.129638] RAX: 0000000000004192 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6476.134805] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6476.148260] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6476.152462] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6476.158335] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6476.168924] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6476.186420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6476.191852] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6476.195780] Call Trace: [ 6476.197349] [ 6476.200009] ? show_regs.cold.9+0x22/0x2f [ 6476.201565] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6476.216183] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6476.221207] ? watchdog+0x40/0x40 [ 6476.224323] ? hrtimer_interrupt+0x128/0x2c0 [ 6476.231324] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6476.240424] ? apic_timer_interrupt+0xf/0x20 [ 6476.241759] [ 6476.242590] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6476.251275] do_raw_spin_lock+0x177/0x190 [ 6476.253842] _raw_spin_lock+0x21/0x30 [ 6476.255482] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6476.258402] ? do_raw_spin_unlock+0x75/0x190 [ 6476.260735] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6476.265383] ? do_wait_intr+0xf0/0xf0 [ 6476.268831] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6476.272499] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6476.276321] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6476.279908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6476.284078] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6476.287698] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6476.290336] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6476.292934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6476.295739] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6476.298860] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6476.301371] lmv_intent_open+0x338/0x1150 [lmv] [ 6476.303963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6476.306770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6476.309490] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6476.311838] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6476.314474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6476.317136] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6476.319887] ? do_raw_spin_unlock+0x75/0x190 [ 6476.322043] ? _raw_spin_unlock+0x12/0x30 [ 6476.323844] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6476.332417] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6476.338362] ll_atomic_open+0x36a/0xd70 [lustre] [ 6476.342341] lookup_open+0x356/0xb40 [ 6476.344086] do_last+0x313/0xfc0 [ 6476.345601] ? nd_jump_root+0xe5/0x160 [ 6476.348475] ? path_init+0x437/0x520 [ 6476.350947] path_openat+0xf7/0x500 [ 6476.352975] ? do_raw_spin_unlock+0x75/0x190 [ 6476.356636] do_filp_open+0x99/0x140 [ 6476.359378] ? getname_flags+0x6e/0x330 [ 6476.360934] ? __check_object_size+0xff/0x256 [ 6476.363065] ? do_raw_spin_unlock+0x75/0x190 [ 6476.364977] ? _raw_spin_unlock+0x12/0x30 [ 6476.367047] do_sys_openat2+0x2b4/0x410 [ 6476.368901] do_sys_open+0x73/0xa0 [ 6476.369982] __x64_sys_openat+0x24/0x30 [ 6476.372075] do_syscall_64+0xc1/0x440 [ 6476.373286] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6476.374835] RIP: 0033:0x7f1a63983332 [ 6476.376230] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6476.383274] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6476.386583] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6476.388901] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6476.394241] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6476.399287] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6476.402265] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6480.865512] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4457s! [ 6480.875031] Showing busy workqueues and worker pools: [ 6480.880853] workqueue events: flags=0x0 [ 6480.882822] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6480.892422] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6480.894752] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6480.904443] pending: sec_gc_main [ptlrpc] [ 6480.910475] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6480.919316] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6480.930853] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6480.936729] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6480.942346] workqueue events_long: flags=0x0 [ 6480.944333] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6480.950503] pending: lu_objects_flush [obdclass] [ 6480.954634] workqueue events_unbound: flags=0x2 [ 6480.957659] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6480.963611] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6480.967838] workqueue events_power_efficient: flags=0x80 [ 6480.972553] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6480.978930] pending: neigh_periodic_work, do_cache_clean [ 6480.984504] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6480.986995] pending: neigh_periodic_work, check_lifetime [ 6480.989977] workqueue mm_percpu_wq: flags=0x8 [ 6480.992982] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6480.995467] pending: vmstat_update [ 6481.002607] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6481.008569] pending: vmstat_update [ 6481.013904] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6481.021928] pending: vmstat_update [ 6481.022925] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6481.026030] pending: vmstat_update [ 6481.029683] workqueue writeback: flags=0x4a [ 6481.032707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6481.035034] pending: wb_workfn, wb_workfn [ 6481.040926] workqueue xprtiod: flags=0xa [ 6481.044508] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6481.046856] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6481.054930] workqueue ipv6_addrconf: flags=0x40008 [ 6481.060096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6481.064847] pending: addrconf_verify_work [ 6481.070531] workqueue knbd0-recv: flags=0x1a [ 6481.072139] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6481.074764] in-flight: 647:recv_work [ 6481.076794] workqueue ptlrpc_pinger: flags=0x2 [ 6481.078627] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6481.080962] pending: ptlrpc_pinger_main [ptlrpc] [ 6481.083300] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6471s workers=2 idle: 652 [ 6484.056012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6484.057116] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6484.058337] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6484.058345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6484.058421] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6484.058426] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6484.058429] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6484.058442] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6484.058446] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6484.058451] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6484.058453] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6484.058456] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6484.058458] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6484.058461] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6484.058464] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6484.058467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6484.058469] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6484.058477] Call Trace: [ 6484.058482] [ 6484.058496] ? show_regs.cold.9+0x22/0x2f [ 6484.058505] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6484.058511] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6484.058516] ? watchdog+0x40/0x40 [ 6484.058520] ? hrtimer_interrupt+0x128/0x2c0 [ 6484.058524] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6484.058530] ? apic_timer_interrupt+0xf/0x20 [ 6484.058534] [ 6484.058537] ? native_safe_halt+0x2e/0x50 [ 6484.058541] kvm_wait+0xcd/0xe0 [ 6484.058547] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6484.058554] do_raw_spin_lock+0x177/0x190 [ 6484.058558] _raw_spin_lock+0x21/0x30 [ 6484.058562] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6484.058882] obd_mod_free+0x46/0x140 [ptlrpc] [ 6484.059772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul [ 6484.059801] mdc_commit_open+0x91/0xb0 [mdc] [ 6484.059832] crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6484.059855] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6484.059861] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6484.059863] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6484.059874] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6484.059878] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6484.059882] RAX: 0000000000002db8 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6484.059884] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6484.059887] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6484.059889] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6484.059891] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6484.059894] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6484.059897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6484.059899] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6484.059907] Call Trace: [ 6484.059910] [ 6484.059914] ? show_regs.cold.9+0x22/0x2f [ 6484.059922] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6484.059927] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6484.059931] ? watchdog+0x40/0x40 [ 6484.059935] ? hrtimer_interrupt+0x128/0x2c0 [ 6484.059939] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6484.059946] ? apic_timer_interrupt+0xf/0x20 [ 6484.059950] [ 6484.059953] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6484.059848] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6484.059959] do_raw_spin_lock+0x177/0x190 [ 6484.059967] _raw_spin_lock+0x21/0x30 [ 6484.060119] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6484.059972] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6484.060523] after_reply+0xe2c/0x1780 [ptlrpc] [ 6484.061177] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6484.061525] ? __next_timer_interrupt+0x160/0x160 [ 6484.061535] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6484.062193] ? do_raw_spin_unlock+0x75/0x190 [ 6484.062201] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6484.066451] ? do_wait_intr+0xf0/0xf0 [ 6484.066475] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6484.066779] kthread+0x1d1/0x200 [ 6484.066785] ? set_kthread_struct+0x70/0x70 [ 6484.066789] ret_from_fork+0x1f/0x30 [ 6484.067020] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6484.067305] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6484.067313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6484.067315] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6484.067327] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6484.067407] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6484.067412] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6484.067415] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6484.067417] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6484.067419] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6484.067422] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6484.067537] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6484.067541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6484.067543] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6484.067551] Call Trace: [ 6484.067555] [ 6484.067559] ? show_regs.cold.9+0x22/0x2f [ 6484.067568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6484.067574] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6484.067578] ? watchdog+0x40/0x40 [ 6484.067582] ? hrtimer_interrupt+0x128/0x2c0 [ 6484.067586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6484.067592] ? apic_timer_interrupt+0xf/0x20 [ 6484.067597] [ 6484.067600] ? native_safe_halt+0x2e/0x50 [ 6484.067603] kvm_wait+0xcd/0xe0 [ 6484.067611] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6484.067618] do_raw_spin_lock+0x177/0x190 [ 6484.067622] _raw_spin_lock+0x21/0x30 [ 6484.067627] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6484.067893] ? do_raw_spin_unlock+0x75/0x190 [ 6484.067899] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6484.071301] ? do_wait_intr+0xf0/0xf0 [ 6484.071322] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6484.071666] mdc_close+0x53b/0xd70 [mdc] [ 6484.071709] lmv_close+0x18d/0x490 [lmv] [ 6484.071745] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6484.071884] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6484.071969] ll_md_close+0x274/0x800 [lustre] [ 6484.073684] ll_file_release+0x128/0xc50 [lustre] [ 6484.073753] __fput+0x10d/0x440 [ 6484.073761] ____fput+0x12/0x20 [ 6484.073765] task_work_run+0xc6/0x110 [ 6484.073771] exit_to_usermode_loop+0x1dd/0x1f0 [ 6484.073776] do_syscall_64+0x3d6/0x440 [ 6484.073780] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6484.073787] RIP: 0033:0x7f1a63983cd8 [ 6484.073793] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6484.073796] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6484.073800] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6484.073801] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6484.073803] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6484.073804] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6484.073806] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6484.882919] ? do_wait_intr+0xf0/0xf0 [ 6484.886037] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6484.889641] mdc_close+0x53b/0xd70 [mdc] [ 6484.893354] lmv_close+0x18d/0x490 [lmv] [ 6484.896781] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6484.901184] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6484.905012] ll_md_close+0x274/0x800 [lustre] [ 6484.909154] ll_file_release+0x128/0xc50 [lustre] [ 6484.915197] __fput+0x10d/0x440 [ 6484.916383] ____fput+0x12/0x20 [ 6484.921888] task_work_run+0xc6/0x110 [ 6484.924735] exit_to_usermode_loop+0x1dd/0x1f0 [ 6484.930667] do_syscall_64+0x3d6/0x440 [ 6484.932145] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6484.939459] RIP: 0033:0x7f1a63983cd8 [ 6484.942779] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6484.958390] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6484.967803] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6484.972806] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6484.983011] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6484.989182] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6484.998305] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6508.062377] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 6508.067637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6508.088350] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6508.092258] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6508.095189] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6508.097631] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6508.102975] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6508.105395] RAX: 0000000000006207 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6508.107580] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6508.110170] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6508.112428] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6508.114704] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6508.116868] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6508.121387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6508.123751] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6508.126684] Call Trace: [ 6508.128306] [ 6508.129174] ? show_regs.cold.9+0x22/0x2f [ 6508.131448] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6508.136455] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6508.141451] ? watchdog+0x40/0x40 [ 6508.145681] ? hrtimer_interrupt+0x128/0x2c0 [ 6508.147455] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6508.152342] ? apic_timer_interrupt+0xf/0x20 [ 6508.155552] [ 6508.158345] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6508.164413] do_raw_spin_lock+0x177/0x190 [ 6508.168958] _raw_spin_lock+0x21/0x30 [ 6508.172005] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6508.174958] ? do_raw_spin_unlock+0x75/0x190 [ 6508.177566] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6508.180916] ? do_wait_intr+0xf0/0xf0 [ 6508.182916] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6508.188147] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6508.195259] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6508.201432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6508.204553] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6508.210643] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6508.215011] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6508.217753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6508.221585] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6508.228516] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6508.232679] lmv_intent_open+0x338/0x1150 [lmv] [ 6508.234454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6508.243191] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6508.247957] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6508.251475] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6508.253679] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6508.256356] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6508.258959] ? do_raw_spin_unlock+0x75/0x190 [ 6508.262199] ? _raw_spin_unlock+0x12/0x30 [ 6508.266401] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6508.269457] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6508.272592] ll_atomic_open+0x36a/0xd70 [lustre] [ 6508.274975] lookup_open+0x356/0xb40 [ 6508.276604] do_last+0x313/0xfc0 [ 6508.278954] ? nd_jump_root+0xe5/0x160 [ 6508.280426] ? path_init+0x437/0x520 [ 6508.283598] path_openat+0xf7/0x500 [ 6508.286203] ? do_raw_spin_unlock+0x75/0x190 [ 6508.288749] do_filp_open+0x99/0x140 [ 6508.290466] ? getname_flags+0x6e/0x330 [ 6508.292564] ? __check_object_size+0xff/0x256 [ 6508.295398] ? do_raw_spin_unlock+0x75/0x190 [ 6508.300315] ? _raw_spin_unlock+0x12/0x30 [ 6508.302357] do_sys_openat2+0x2b4/0x410 [ 6508.304777] do_sys_open+0x73/0xa0 [ 6508.306862] __x64_sys_openat+0x24/0x30 [ 6508.309555] do_syscall_64+0xc1/0x440 [ 6508.318461] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6508.320995] RIP: 0033:0x7f1a63983332 [ 6508.324926] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6508.337959] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6508.342895] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6508.345588] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6508.352176] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6508.357330] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6508.360469] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6511.585037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4488s! [ 6511.590148] Showing busy workqueues and worker pools: [ 6511.592990] workqueue events: flags=0x0 [ 6511.595291] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6511.598781] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6511.602568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6511.606575] pending: sec_gc_main [ptlrpc] [ 6511.609201] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6511.611790] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6511.616793] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6511.620045] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6511.624804] workqueue events_long: flags=0x0 [ 6511.627158] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6511.630627] pending: lu_objects_flush [obdclass] [ 6511.632890] workqueue events_unbound: flags=0x2 [ 6511.635191] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6511.638431] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6511.643343] workqueue events_power_efficient: flags=0x80 [ 6511.645906] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6511.647944] pending: neigh_periodic_work, do_cache_clean [ 6511.650196] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6511.652628] pending: neigh_periodic_work, check_lifetime [ 6511.654219] workqueue mm_percpu_wq: flags=0x8 [ 6511.656218] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6511.659027] pending: vmstat_update [ 6511.660694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6511.663513] pending: vmstat_update [ 6511.664903] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6511.667045] pending: vmstat_update [ 6511.668733] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6511.671417] pending: vmstat_update [ 6511.673167] workqueue writeback: flags=0x4a [ 6511.674791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6511.677677] pending: wb_workfn, wb_workfn [ 6511.680679] workqueue xprtiod: flags=0xa [ 6511.682337] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6511.686250] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6511.694130] workqueue ipv6_addrconf: flags=0x40008 [ 6511.699127] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6511.705585] pending: addrconf_verify_work [ 6511.709079] workqueue knbd0-recv: flags=0x1a [ 6511.712662] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6511.719302] in-flight: 647:recv_work [ 6511.722886] workqueue ptlrpc_pinger: flags=0x2 [ 6511.727264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6511.733180] pending: ptlrpc_pinger_main [ptlrpc] [ 6511.739906] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6501s workers=2 idle: 652 [ 6512.056212] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6512.057183] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6512.061364] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6512.061377] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6512.061451] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6512.061467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6512.061474] Modules linked in: lustre(O) [ 6512.061471] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6512.061481] osp(O) ofd(O) lod(O) [ 6512.061488] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6512.061491] mdt(O) mdd(O) [ 6512.061499] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 [ 6512.061500] mgs(O) lquota(O) [ 6512.061505] ORIG_RAX: ffffffffffffff13 [ 6512.061507] lfsck(O) [ 6512.061511] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6512.061512] obdecho(O) mgc(O) [ 6512.061518] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6512.061519] mdc(O) lov(O) [ 6512.061526] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6512.061529] osc(O) [ 6512.061533] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6512.061534] lmv(O) ec(O) [ 6512.061540] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6512.061543] fid(O) fld(O) [ 6512.061548] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6512.061551] ptlrpc_gss(O) ptlrpc(O) [ 6512.061556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6512.061560] obdclass(O) [ 6512.061564] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6512.061565] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [ 6512.061577] Call Trace: [ 6512.061578] intel_rapl_msr intel_rapl_common sb_edac [ 6512.061587] [ 6512.061587] rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel [ 6512.061598] ? show_regs.cold.9+0x22/0x2f [ 6512.061607] ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [ 6512.061614] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6512.061621] dm_mod sha512_ssse3 sha512_generic [ 6512.061628] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6512.061632] [ 6512.061636] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6512.061637] ? watchdog+0x40/0x40 [ 6512.061646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6512.061648] ? hrtimer_interrupt+0x128/0x2c0 [ 6512.061653] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6512.061659] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6512.061666] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6512.061674] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 [ 6512.061673] ? apic_timer_interrupt+0xf/0x20 [ 6512.061679] ORIG_RAX: ffffffffffffff13 [ 6512.061683] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6512.061685] [ 6512.061688] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6512.061692] ? native_safe_halt+0x2e/0x50 [ 6512.061694] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6512.061701] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6512.061705] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6512.061701] kvm_wait+0xcd/0xe0 [ 6512.061709] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6512.061716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6512.061715] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6512.061719] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6512.061729] Call Trace: [ 6512.061735] [ 6512.061738] ? show_regs.cold.9+0x22/0x2f [ 6512.061728] do_raw_spin_lock+0x177/0x190 [ 6512.061747] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6512.061751] _raw_spin_lock+0x21/0x30 [ 6512.061760] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6512.061779] ? watchdog+0x40/0x40 [ 6512.061784] ? hrtimer_interrupt+0x128/0x2c0 [ 6512.061788] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6512.061794] ? apic_timer_interrupt+0xf/0x20 [ 6512.061799] [ 6512.061802] ? native_safe_halt+0x2e/0x50 [ 6512.061806] kvm_wait+0xcd/0xe0 [ 6512.061811] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6512.061818] do_raw_spin_lock+0x177/0x190 [ 6512.061822] _raw_spin_lock+0x21/0x30 [ 6512.061826] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6512.062463] ? do_raw_spin_unlock+0x75/0x190 [ 6512.061772] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6512.062904] obd_mod_free+0x46/0x140 [ptlrpc] [ 6512.062469] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6512.063654] ? do_wait_intr+0xf0/0xf0 [ 6512.063799] mdc_commit_open+0x91/0xb0 [mdc] [ 6512.063868] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6512.063663] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6512.065244] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6512.066071] after_reply+0xe2c/0x1780 [ptlrpc] [ 6512.066262] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6512.066655] ? __next_timer_interrupt+0x160/0x160 [ 6512.066665] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6512.066936] ? do_wait_intr+0xf0/0xf0 [ 6512.066946] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6512.067140] kthread+0x1d1/0x200 [ 6512.067146] ? set_kthread_struct+0x70/0x70 [ 6512.067150] ret_from_fork+0x1f/0x30 [ 6512.068320] Modules linked in: lustre(O) [ 6512.071193] mdc_close+0x53b/0xd70 [mdc] [ 6512.089154] osp(O) [ 6512.092743] lmv_close+0x18d/0x490 [lmv] [ 6512.095702] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6512.095829] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6512.095835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6512.095838] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6512.095851] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6512.095855] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6512.095861] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6512.095863] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6512.095865] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6512.095867] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6512.095870] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6512.095873] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6512.095876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6512.095878] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6512.095887] Call Trace: [ 6512.095892] [ 6512.095897] ? show_regs.cold.9+0x22/0x2f [ 6512.095906] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6512.095912] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6512.095917] ? watchdog+0x40/0x40 [ 6512.095921] ? hrtimer_interrupt+0x128/0x2c0 [ 6512.095925] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6512.095931] ? apic_timer_interrupt+0xf/0x20 [ 6512.095936] [ 6512.095939] ? native_safe_halt+0x2e/0x50 [ 6512.095942] kvm_wait+0xcd/0xe0 [ 6512.095949] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6512.095957] do_raw_spin_lock+0x177/0x190 [ 6512.095962] _raw_spin_lock+0x21/0x30 [ 6512.095966] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6512.096728] ? do_raw_spin_unlock+0x75/0x190 [ 6512.096734] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6512.097052] ? do_wait_intr+0xf0/0xf0 [ 6512.097062] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6512.097258] mdc_close+0x53b/0xd70 [mdc] [ 6512.097297] lmv_close+0x18d/0x490 [lmv] [ 6512.097328] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6512.097628] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6512.097710] ll_md_close+0x274/0x800 [lustre] [ 6512.097914] ll_file_release+0x128/0xc50 [lustre] [ 6512.097994] __fput+0x10d/0x440 [ 6512.101948] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6512.103474] ____fput+0x12/0x20 [ 6512.103487] task_work_run+0xc6/0x110 [ 6512.103494] exit_to_usermode_loop+0x1dd/0x1f0 [ 6512.103502] do_syscall_64+0x3d6/0x440 [ 6512.103507] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6512.103515] RIP: 0033:0x7f1a63983cd8 [ 6512.103522] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6512.103526] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6512.103530] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6512.103532] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6512.103534] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6512.103537] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6512.103538] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6512.491275] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6512.493238] ll_md_close+0x274/0x800 [lustre] [ 6512.495692] ll_file_release+0x128/0xc50 [lustre] [ 6512.498464] __fput+0x10d/0x440 [ 6512.500304] ____fput+0x12/0x20 [ 6512.503725] task_work_run+0xc6/0x110 [ 6512.506084] exit_to_usermode_loop+0x1dd/0x1f0 [ 6512.509492] do_syscall_64+0x3d6/0x440 [ 6512.511458] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6512.514201] RIP: 0033:0x7f1a63983cd8 [ 6512.518252] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6512.536480] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6512.541225] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6512.545747] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6512.549636] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6512.552737] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6512.556304] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6536.055581] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 6536.060830] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6536.086772] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6536.092215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6536.095163] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6536.096807] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6536.107875] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6536.113490] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6536.119349] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6536.122943] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6536.126792] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6536.129770] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6536.134298] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6536.137375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6536.140812] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6536.143780] Call Trace: [ 6536.145824] [ 6536.146431] ? show_regs.cold.9+0x22/0x2f [ 6536.149626] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6536.153202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6536.156603] ? watchdog+0x40/0x40 [ 6536.158583] ? hrtimer_interrupt+0x128/0x2c0 [ 6536.160903] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6536.163817] ? apic_timer_interrupt+0xf/0x20 [ 6536.166824] [ 6536.168174] ? native_safe_halt+0x2e/0x50 [ 6536.170453] kvm_wait+0xcd/0xe0 [ 6536.171860] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6536.175458] do_raw_spin_lock+0x177/0x190 [ 6536.177630] _raw_spin_lock+0x21/0x30 [ 6536.179851] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6536.184413] ? do_raw_spin_unlock+0x75/0x190 [ 6536.186895] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6536.190828] ? do_wait_intr+0xf0/0xf0 [ 6536.193093] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6536.196160] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6536.198590] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6536.202078] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6536.205410] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6536.209132] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6536.212170] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6536.215038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6536.218848] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6536.222728] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6536.227170] lmv_intent_open+0x338/0x1150 [lmv] [ 6536.230125] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6536.234777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6536.237817] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6536.239857] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6536.243359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6536.245896] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6536.248839] ? do_raw_spin_unlock+0x75/0x190 [ 6536.251481] ? _raw_spin_unlock+0x12/0x30 [ 6536.253872] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6536.257130] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6536.260299] ll_atomic_open+0x36a/0xd70 [lustre] [ 6536.263282] lookup_open+0x356/0xb40 [ 6536.264981] do_last+0x313/0xfc0 [ 6536.266689] ? nd_jump_root+0xe5/0x160 [ 6536.268327] ? path_init+0x437/0x520 [ 6536.270123] path_openat+0xf7/0x500 [ 6536.271813] ? do_raw_spin_unlock+0x75/0x190 [ 6536.274198] do_filp_open+0x99/0x140 [ 6536.276695] ? getname_flags+0x6e/0x330 [ 6536.278365] ? __check_object_size+0xff/0x256 [ 6536.280353] ? do_raw_spin_unlock+0x75/0x190 [ 6536.284164] ? _raw_spin_unlock+0x12/0x30 [ 6536.285403] do_sys_openat2+0x2b4/0x410 [ 6536.286437] do_sys_open+0x73/0xa0 [ 6536.287329] __x64_sys_openat+0x24/0x30 [ 6536.289373] do_syscall_64+0xc1/0x440 [ 6536.290436] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6536.292401] RIP: 0033:0x7f1a63983332 [ 6536.293337] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6536.298507] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6536.300488] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6536.302329] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6536.304133] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6536.306011] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6536.308065] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6540.057257] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6540.057564] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6540.059037] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6540.059047] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6540.059120] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6540.059126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6540.059128] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6540.059143] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6540.059147] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6540.059151] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6540.059154] RDX: 0000000000004c78 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6540.059156] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6540.059158] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6540.059160] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6540.059163] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6540.059166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6540.059168] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6540.059180] Call Trace: [ 6540.059185] [ 6540.059191] ? show_regs.cold.9+0x22/0x2f [ 6540.059200] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6540.059207] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6540.059211] ? watchdog+0x40/0x40 [ 6540.059215] ? hrtimer_interrupt+0x128/0x2c0 [ 6540.059219] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6540.059229] ? apic_timer_interrupt+0xf/0x20 [ 6540.059235] [ 6540.059238] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6540.059244] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6540.059249] do_raw_spin_lock+0x177/0x190 [ 6540.059254] _raw_spin_lock+0x21/0x30 [ 6540.059259] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6540.059578] obd_mod_free+0x46/0x140 [ptlrpc] [ 6540.059855] mdc_commit_open+0x91/0xb0 [mdc] [ 6540.059890] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6540.060096] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6540.060304] after_reply+0xe2c/0x1780 [ptlrpc] [ 6540.061618] Modules linked in: [ 6540.062049] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6540.062463] ? __next_timer_interrupt+0x160/0x160 [ 6540.062474] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6540.062705] ? do_wait_intr+0xf0/0xf0 [ 6540.062714] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6540.064300] kthread+0x1d1/0x200 [ 6540.064308] ? set_kthread_struct+0x70/0x70 [ 6540.064312] ret_from_fork+0x1f/0x30 [ 6540.065758] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6540.065835] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6540.065842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6540.065845] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6540.065856] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6540.065860] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6540.065864] RAX: 0000000000007d09 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6540.065866] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6540.065868] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6540.065870] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6540.065872] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6540.065876] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6540.065879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6540.065881] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6540.065889] Call Trace: [ 6540.065892] [ 6540.065896] ? show_regs.cold.9+0x22/0x2f [ 6540.065904] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6540.065910] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6540.065913] ? watchdog+0x40/0x40 [ 6540.065917] ? hrtimer_interrupt+0x128/0x2c0 [ 6540.065921] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6540.065928] ? apic_timer_interrupt+0xf/0x20 [ 6540.065932] [ 6540.065934] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6540.065940] do_raw_spin_lock+0x177/0x190 [ 6540.065997] _raw_spin_lock+0x21/0x30 [ 6540.066002] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6540.066262] ? do_raw_spin_unlock+0x75/0x190 [ 6540.066268] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6540.066623] ? do_wait_intr+0xf0/0xf0 [ 6540.066631] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6540.066827] mdc_close+0x53b/0xd70 [mdc] [ 6540.066860] lmv_close+0x18d/0x490 [lmv] [ 6540.066889] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6540.067328] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6540.067408] ll_md_close+0x274/0x800 [lustre] [ 6540.067620] ll_file_release+0x128/0xc50 [lustre] [ 6540.067700] __fput+0x10d/0x440 [ 6540.067708] ____fput+0x12/0x20 [ 6540.067712] task_work_run+0xc6/0x110 [ 6540.067719] exit_to_usermode_loop+0x1dd/0x1f0 [ 6540.067726] do_syscall_64+0x3d6/0x440 [ 6540.067730] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6540.067737] RIP: 0033:0x7f1a63983cd8 [ 6540.067744] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6540.067748] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6540.067752] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6540.067754] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6540.067757] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6540.067759] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6540.067761] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6540.507249] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6540.523386] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6540.527599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6540.530958] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6540.533010] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6540.539800] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6540.542323] RAX: 0000000000007e0d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6540.545030] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6540.547970] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6540.556140] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6540.563763] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6540.573423] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6540.580257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6540.583471] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6540.587928] Call Trace: [ 6540.589454] [ 6540.590395] ? show_regs.cold.9+0x22/0x2f [ 6540.592584] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6540.594628] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6540.596176] ? watchdog+0x40/0x40 [ 6540.596900] ? hrtimer_interrupt+0x128/0x2c0 [ 6540.598560] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6540.599762] ? apic_timer_interrupt+0xf/0x20 [ 6540.600790] [ 6540.601410] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6540.603291] do_raw_spin_lock+0x177/0x190 [ 6540.604472] _raw_spin_lock+0x21/0x30 [ 6540.605359] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6540.606963] ? do_raw_spin_unlock+0x75/0x190 [ 6540.608213] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6540.609512] ? do_wait_intr+0xf0/0xf0 [ 6540.610706] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6540.612727] mdc_close+0x53b/0xd70 [mdc] [ 6540.613936] lmv_close+0x18d/0x490 [lmv] [ 6540.615416] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6540.617478] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6540.619422] ll_md_close+0x274/0x800 [lustre] [ 6540.620638] ll_file_release+0x128/0xc50 [lustre] [ 6540.621765] __fput+0x10d/0x440 [ 6540.622450] ____fput+0x12/0x20 [ 6540.623195] task_work_run+0xc6/0x110 [ 6540.624754] exit_to_usermode_loop+0x1dd/0x1f0 [ 6540.626365] do_syscall_64+0x3d6/0x440 [ 6540.628210] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6540.630123] RIP: 0033:0x7f1a63983cd8 [ 6540.631675] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6540.638562] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6540.642790] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6540.647054] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6540.651436] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6540.654956] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6540.657937] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6541.794041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4518s! [ 6541.797845] Showing busy workqueues and worker pools: [ 6541.801874] workqueue events: flags=0x0 [ 6541.803385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6541.806418] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6541.809978] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6541.812353] pending: sec_gc_main [ptlrpc] [ 6541.817591] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6541.826115] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6541.833864] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6541.838826] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6541.849834] workqueue events_long: flags=0x0 [ 6541.852574] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6541.855764] pending: lu_objects_flush [obdclass] [ 6541.859135] workqueue events_unbound: flags=0x2 [ 6541.861766] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6541.864437] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6541.866982] workqueue events_power_efficient: flags=0x80 [ 6541.868306] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6541.870322] pending: neigh_periodic_work, do_cache_clean [ 6541.871890] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6541.874013] pending: neigh_periodic_work, check_lifetime [ 6541.875503] workqueue mm_percpu_wq: flags=0x8 [ 6541.876569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6541.878204] pending: vmstat_update [ 6541.879172] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6541.880951] pending: vmstat_update [ 6541.881871] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6541.883365] pending: vmstat_update [ 6541.884631] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6541.886291] pending: vmstat_update [ 6541.887753] workqueue writeback: flags=0x4a [ 6541.888842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6541.890595] pending: wb_workfn, wb_workfn [ 6541.891760] workqueue xprtiod: flags=0xa [ 6541.892698] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6541.894216] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6541.896796] workqueue ipv6_addrconf: flags=0x40008 [ 6541.898265] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6541.899850] pending: addrconf_verify_work [ 6541.900890] workqueue knbd0-recv: flags=0x1a [ 6541.901979] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6541.905325] in-flight: 647:recv_work [ 6541.906899] workqueue ptlrpc_pinger: flags=0x2 [ 6541.908758] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6541.911334] pending: ptlrpc_pinger_main [ptlrpc] [ 6541.913956] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6531s workers=2 idle: 652 [ 6568.053463] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6568.056974] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6568.057020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6568.057027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6568.057172] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6568.057177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6568.057180] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6568.057276] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6568.057280] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6568.057285] RAX: 0000000000005a85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6568.057287] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6568.057289] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6568.057291] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6568.057294] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6568.057301] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6568.057304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6568.057306] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6568.057314] Call Trace: [ 6568.057327] [ 6568.057334] ? show_regs.cold.9+0x22/0x2f [ 6568.057345] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6568.057351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6568.057355] ? watchdog+0x40/0x40 [ 6568.057361] ? hrtimer_interrupt+0x128/0x2c0 [ 6568.057365] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6568.057376] ? apic_timer_interrupt+0xf/0x20 [ 6568.057380] [ 6568.057382] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6568.057388] do_raw_spin_lock+0x177/0x190 [ 6568.057392] _raw_spin_lock+0x21/0x30 [ 6568.057399] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6568.057664] ? do_raw_spin_unlock+0x75/0x190 [ 6568.057669] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6568.057868] ? do_wait_intr+0xf0/0xf0 [ 6568.057878] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6568.058031] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6568.058037] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6568.058104] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6568.058108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6568.058110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6568.058119] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6568.058123] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6568.058127] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6568.058129] RDX: 0000000000002a92 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6568.058132] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6568.058134] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6568.058137] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6568.058139] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6568.058143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6568.058079] mdc_close+0x53b/0xd70 [mdc] [ 6568.058147] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6568.058156] Call Trace: [ 6568.058213] [ 6568.058218] ? show_regs.cold.9+0x22/0x2f [ 6568.058202] lmv_close+0x18d/0x490 [lmv] [ 6568.058238] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6568.058250] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6568.058258] ? watchdog+0x40/0x40 [ 6568.058263] ? hrtimer_interrupt+0x128/0x2c0 [ 6568.058267] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6568.058275] ? apic_timer_interrupt+0xf/0x20 [ 6568.058280] [ 6568.058284] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6568.058289] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6568.058294] do_raw_spin_lock+0x177/0x190 [ 6568.058299] _raw_spin_lock+0x21/0x30 [ 6568.058252] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6568.058529] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6568.058558] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6568.058598] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6568.058604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6568.058606] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6568.058613] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6568.058617] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6568.058621] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6568.058623] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6568.058625] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6568.058628] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6568.058630] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6568.058632] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6568.058636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6568.058638] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6568.058645] Call Trace: [ 6568.058648] [ 6568.058652] ? show_regs.cold.9+0x22/0x2f [ 6568.058658] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6568.058663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6568.058304] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6568.058667] ? watchdog+0x40/0x40 [ 6568.058882] ? hrtimer_interrupt+0x128/0x2c0 [ 6568.058887] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6568.058893] ? apic_timer_interrupt+0xf/0x20 [ 6568.058898] [ 6568.058901] ? native_safe_halt+0x2e/0x50 [ 6568.058905] kvm_wait+0xcd/0xe0 [ 6568.058913] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6568.058920] do_raw_spin_lock+0x177/0x190 [ 6568.058924] _raw_spin_lock+0x21/0x30 [ 6568.058839] ll_md_close+0x274/0x800 [lustre] [ 6568.059522] ll_file_release+0x128/0xc50 [lustre] [ 6568.059603] __fput+0x10d/0x440 [ 6568.059667] ____fput+0x12/0x20 [ 6568.059671] task_work_run+0xc6/0x110 [ 6568.059680] exit_to_usermode_loop+0x1dd/0x1f0 [ 6568.059688] do_syscall_64+0x3d6/0x440 [ 6568.059692] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6568.059699] RIP: 0033:0x7f1a63983cd8 [ 6568.059718] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6568.058928] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6568.059728] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6568.059732] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6568.059734] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6568.059737] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6568.059798] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6568.059800] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6568.059156] obd_mod_free+0x46/0x140 [ptlrpc] [ 6568.060076] mdc_commit_open+0x91/0xb0 [mdc] [ 6568.060109] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6568.060423] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6568.060714] after_reply+0xe2c/0x1780 [ptlrpc] [ 6568.061022] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6568.061288] ? __next_timer_interrupt+0x160/0x160 [ 6568.061297] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6568.061838] ? do_wait_intr+0xf0/0xf0 [ 6568.061847] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6568.062369] kthread+0x1d1/0x200 [ 6568.062376] ? set_kthread_struct+0x70/0x70 [ 6568.062380] ret_from_fork+0x1f/0x30 [ 6568.066681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6568.066768] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6568.066775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6568.066778] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6568.066792] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6568.066800] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6568.066805] RAX: 0000000000007915 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6568.066808] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6568.066810] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6568.066812] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6568.066815] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6568.066818] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6568.066821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6568.066824] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6568.066832] Call Trace: [ 6568.066837] [ 6568.066843] ? show_regs.cold.9+0x22/0x2f [ 6568.066852] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6568.066865] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6568.066870] ? watchdog+0x40/0x40 [ 6568.066874] ? hrtimer_interrupt+0x128/0x2c0 [ 6568.066879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6568.066887] ? apic_timer_interrupt+0xf/0x20 [ 6568.066892] [ 6568.069042] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6568.071044] ? do_raw_spin_unlock+0x75/0x190 [ 6568.071061] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6568.071326] ? do_wait_intr+0xf0/0xf0 [ 6568.071335] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6568.071531] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6568.071761] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6568.071968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6568.072077] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6568.072113] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6568.072138] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6568.072163] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6568.072257] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6568.072465] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6568.072492] lmv_intent_open+0x338/0x1150 [lmv] [ 6568.072519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6568.072602] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6568.072685] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6568.072707] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6568.072790] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6568.072872] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6568.072956] ? do_raw_spin_unlock+0x75/0x190 [ 6568.072962] ? _raw_spin_unlock+0x12/0x30 [ 6568.072969] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6568.072995] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6568.073015] ll_atomic_open+0x36a/0xd70 [lustre] [ 6568.073098] lookup_open+0x356/0xb40 [ 6568.073106] do_last+0x313/0xfc0 [ 6568.073110] ? nd_jump_root+0xe5/0x160 [ 6568.073114] ? path_init+0x437/0x520 [ 6568.073118] path_openat+0xf7/0x500 [ 6568.073122] ? do_raw_spin_unlock+0x75/0x190 [ 6568.073126] do_filp_open+0x99/0x140 [ 6568.073131] ? getname_flags+0x6e/0x330 [ 6568.073135] ? __check_object_size+0xff/0x256 [ 6568.073141] ? do_raw_spin_unlock+0x75/0x190 [ 6568.073144] ? _raw_spin_unlock+0x12/0x30 [ 6568.073148] do_sys_openat2+0x2b4/0x410 [ 6568.073151] do_sys_open+0x73/0xa0 [ 6568.073154] __x64_sys_openat+0x24/0x30 [ 6568.073158] do_syscall_64+0xc1/0x440 [ 6568.073163] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6568.073169] RIP: 0033:0x7f1a63983332 [ 6568.073177] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6568.073181] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6568.073186] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6568.073188] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6568.073190] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6568.073192] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6568.073193] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6568.927983] do_raw_spin_lock+0x177/0x190 [ 6568.933461] _raw_spin_lock+0x21/0x30 [ 6568.939443] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6568.947984] ? do_raw_spin_unlock+0x75/0x190 [ 6568.955304] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6568.964255] ? do_wait_intr+0xf0/0xf0 [ 6568.965654] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6568.970015] mdc_close+0x53b/0xd70 [mdc] [ 6568.976429] lmv_close+0x18d/0x490 [lmv] [ 6568.981341] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6568.986177] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6568.993024] ll_md_close+0x274/0x800 [lustre] [ 6568.996642] ll_file_release+0x128/0xc50 [lustre] [ 6568.999370] __fput+0x10d/0x440 [ 6569.005884] ____fput+0x12/0x20 [ 6569.008860] task_work_run+0xc6/0x110 [ 6569.010990] exit_to_usermode_loop+0x1dd/0x1f0 [ 6569.018854] do_syscall_64+0x3d6/0x440 [ 6569.021118] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6569.024621] RIP: 0033:0x7f1a63983cd8 [ 6569.027159] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6569.037783] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6569.040629] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6569.044268] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6569.047739] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6569.050318] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6569.053136] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6572.000262] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4548s! [ 6572.010610] Showing busy workqueues and worker pools: [ 6572.017083] workqueue events: flags=0x0 [ 6572.022959] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6572.028545] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6572.033924] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6572.040028] pending: sec_gc_main [ptlrpc] [ 6572.042720] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6572.046237] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6572.052923] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6572.057163] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6572.062380] workqueue events_long: flags=0x0 [ 6572.064769] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6572.069545] pending: lu_objects_flush [obdclass] [ 6572.072791] workqueue events_unbound: flags=0x2 [ 6572.074538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6572.089614] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6572.103429] workqueue events_power_efficient: flags=0x80 [ 6572.105917] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6572.111863] pending: neigh_periodic_work, do_cache_clean [ 6572.119220] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6572.124509] pending: neigh_periodic_work, check_lifetime [ 6572.131014] workqueue mm_percpu_wq: flags=0x8 [ 6572.134975] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6572.143901] pending: vmstat_update [ 6572.151368] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6572.156958] pending: vmstat_update [ 6572.163918] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6572.170933] pending: vmstat_update [ 6572.175653] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6572.179924] pending: vmstat_update [ 6572.184869] workqueue writeback: flags=0x4a [ 6572.190940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6572.195694] pending: wb_workfn, wb_workfn [ 6572.198748] workqueue xprtiod: flags=0xa [ 6572.200948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6572.205709] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6572.214099] workqueue ipv6_addrconf: flags=0x40008 [ 6572.217633] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6572.221988] pending: addrconf_verify_work [ 6572.223970] workqueue knbd0-recv: flags=0x1a [ 6572.227467] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6572.232975] in-flight: 647:recv_work [ 6572.237557] workqueue ptlrpc_pinger: flags=0x2 [ 6572.240988] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6572.249780] pending: ptlrpc_pinger_main [ptlrpc] [ 6572.253089] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6562s workers=2 idle: 652 [ 6582.513214] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 6582.516542] rcu: 3-...!: (2812600 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 6582.526727] (detected by 1, t=4560107 jiffies, g=269633, q=1792) [ 6582.535516] Sending NMI from CPU 1 to CPUs 3: [ 6582.542498] NMI backtrace for cpu 3 [ 6582.542506] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6582.542509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6582.542511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6582.542512] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6582.542515] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 6582.542517] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6582.542518] RDX: 000000000000471b RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6582.542520] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6582.542521] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6582.542522] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6582.542523] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6582.542524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6582.542525] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6582.542526] Call Trace: [ 6582.542527] [ 6582.542528] ? show_regs.cold.9+0x22/0x2f [ 6582.542529] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6582.542530] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6582.542531] ? nmi_handle+0xbd/0x210 [ 6582.542532] ? default_do_nmi+0x4b/0x1b0 [ 6582.542532] ? do_nmi+0x280/0x380 [ 6582.542533] ? end_repeat_nmi+0x16/0x1f [ 6582.542534] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6582.542535] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6582.542536] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6582.542537] [ 6582.542538] do_raw_spin_lock+0x177/0x190 [ 6582.542539] _raw_spin_lock+0x21/0x30 [ 6582.542540] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6582.542541] obd_mod_free+0x46/0x140 [ptlrpc] [ 6582.542541] mdc_commit_open+0x91/0xb0 [mdc] [ 6582.542543] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6582.542544] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6582.542545] after_reply+0xe2c/0x1780 [ptlrpc] [ 6582.542546] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6582.542547] ? __next_timer_interrupt+0x160/0x160 [ 6582.542548] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6582.542549] ? do_wait_intr+0xf0/0xf0 [ 6582.542550] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6582.542551] kthread+0x1d1/0x200 [ 6582.542552] ? set_kthread_struct+0x70/0x70 [ 6582.542553] ret_from_fork+0x1f/0x30 [ 6582.543761] rcu: rcu_sched kthread starved for 4559753 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 6582.767559] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6582.772636] rcu: RCU grace-period kthread stack dump: [ 6582.775674] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6582.782336] Call Trace: [ 6582.783430] __schedule+0x351/0xcb0 [ 6582.787847] schedule+0xc0/0x180 [ 6582.789068] schedule_timeout+0xb4/0x190 [ 6582.790450] ? do_raw_spin_unlock+0x75/0x190 [ 6582.796817] ? __next_timer_interrupt+0x160/0x160 [ 6582.804806] ? prepare_to_swait_event+0xf3/0x140 [ 6582.810727] rcu_gp_kthread+0x8dd/0x15b0 [ 6582.819634] ? force_qs_rnp+0x2c0/0x2c0 [ 6582.821143] kthread+0x1d1/0x200 [ 6582.830312] ? set_kthread_struct+0x70/0x70 [ 6582.832903] ret_from_fork+0x1f/0x30 [ 6582.840001] rcu: Stack dump where RCU GP kthread last ran: [ 6582.842792] Sending NMI from CPU 1 to CPUs 3: [ 6582.852547] NMI backtrace for cpu 3 [ 6582.852554] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6582.852556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6582.852558] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6582.852560] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6582.852562] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 6582.852565] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6582.852566] RDX: 000000000000082f RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6582.852568] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6582.852569] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6582.852570] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6582.852572] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6582.852574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6582.852575] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6582.852577] Call Trace: [ 6582.852578] [ 6582.852579] ? show_regs.cold.9+0x22/0x2f [ 6582.852580] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6582.852581] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6582.852583] ? nmi_handle+0xbd/0x210 [ 6582.852584] ? default_do_nmi+0x4b/0x1b0 [ 6582.852585] ? do_nmi+0x280/0x380 [ 6582.852586] ? end_repeat_nmi+0x16/0x1f [ 6582.852587] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6582.852588] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6582.852589] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6582.852591] [ 6582.852592] do_raw_spin_lock+0x177/0x190 [ 6582.852593] _raw_spin_lock+0x21/0x30 [ 6582.852594] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6582.852595] obd_mod_free+0x46/0x140 [ptlrpc] [ 6582.852597] mdc_commit_open+0x91/0xb0 [mdc] [ 6582.852598] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6582.852599] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6582.852600] after_reply+0xe2c/0x1780 [ptlrpc] [ 6582.852601] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6582.852603] ? __next_timer_interrupt+0x160/0x160 [ 6582.852604] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6582.852605] ? do_wait_intr+0xf0/0xf0 [ 6582.852606] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6582.852607] kthread+0x1d1/0x200 [ 6582.852608] ? set_kthread_struct+0x70/0x70 [ 6582.852610] ret_from_fork+0x1f/0x30 [ 6596.058254] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 6596.069309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6596.100795] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6596.117417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6596.124790] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6596.128664] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6596.135868] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6596.139400] RAX: 0000000000007c0d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6596.145556] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6596.147532] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6596.153923] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6596.156625] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6596.162670] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6596.168122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6596.170723] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6596.175384] Call Trace: [ 6596.177167] [ 6596.178637] ? show_regs.cold.9+0x22/0x2f [ 6596.180084] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6596.182744] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6596.185287] ? watchdog+0x40/0x40 [ 6596.186709] ? hrtimer_interrupt+0x128/0x2c0 [ 6596.187904] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6596.191252] ? apic_timer_interrupt+0xf/0x20 [ 6596.192794] [ 6596.196236] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6596.201078] do_raw_spin_lock+0x177/0x190 [ 6596.203632] _raw_spin_lock+0x21/0x30 [ 6596.205359] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6596.211536] ? do_raw_spin_unlock+0x75/0x190 [ 6596.212780] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6596.217722] ? do_wait_intr+0xf0/0xf0 [ 6596.219132] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6596.222417] mdc_close+0x53b/0xd70 [mdc] [ 6596.225932] lmv_close+0x18d/0x490 [lmv] [ 6596.231218] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6596.234817] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6596.237395] ll_md_close+0x274/0x800 [lustre] [ 6596.241216] ll_file_release+0x128/0xc50 [lustre] [ 6596.247059] __fput+0x10d/0x440 [ 6596.251102] ____fput+0x12/0x20 [ 6596.253727] task_work_run+0xc6/0x110 [ 6596.254890] exit_to_usermode_loop+0x1dd/0x1f0 [ 6596.257252] do_syscall_64+0x3d6/0x440 [ 6596.258981] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6596.261402] RIP: 0033:0x7f1a63983cd8 [ 6596.262994] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6596.271602] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6596.274550] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6596.277259] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6596.280975] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6596.284219] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6596.287320] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6600.053107] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6600.059429] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6600.083895] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6600.095727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6600.102474] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6600.108635] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6600.116438] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6600.118782] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6600.123232] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6600.126648] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6600.129724] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6600.132766] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6600.136054] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6600.139213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6600.141659] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6600.149567] Call Trace: [ 6600.150842] [ 6600.152179] ? show_regs.cold.9+0x22/0x2f [ 6600.154227] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6600.156471] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6600.158539] ? watchdog+0x40/0x40 [ 6600.160876] ? hrtimer_interrupt+0x128/0x2c0 [ 6600.165825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6600.169313] ? apic_timer_interrupt+0xf/0x20 [ 6600.172375] [ 6600.173537] ? native_safe_halt+0x2e/0x50 [ 6600.176520] kvm_wait+0xcd/0xe0 [ 6600.178122] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6600.180159] do_raw_spin_lock+0x177/0x190 [ 6600.182694] _raw_spin_lock+0x21/0x30 [ 6600.185089] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6600.188217] ? do_raw_spin_unlock+0x75/0x190 [ 6600.191446] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6600.194092] ? do_wait_intr+0xf0/0xf0 [ 6600.196593] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6600.199016] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6600.201533] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6600.204195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6600.206697] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6600.209106] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6600.211174] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6600.212959] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6600.215340] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6600.218342] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6600.220812] lmv_intent_open+0x338/0x1150 [lmv] [ 6600.223167] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6600.226465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6600.228737] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6600.233167] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6600.235967] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6600.237767] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6600.241538] ? do_raw_spin_unlock+0x75/0x190 [ 6600.245301] ? _raw_spin_unlock+0x12/0x30 [ 6600.247003] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6600.249343] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6600.251727] ll_atomic_open+0x36a/0xd70 [lustre] [ 6600.253982] lookup_open+0x356/0xb40 [ 6600.255437] do_last+0x313/0xfc0 [ 6600.256725] ? nd_jump_root+0xe5/0x160 [ 6600.258364] ? path_init+0x437/0x520 [ 6600.260732] path_openat+0xf7/0x500 [ 6600.262321] ? do_raw_spin_unlock+0x75/0x190 [ 6600.266460] do_filp_open+0x99/0x140 [ 6600.269495] ? getname_flags+0x6e/0x330 [ 6600.271308] ? __check_object_size+0xff/0x256 [ 6600.273592] ? do_raw_spin_unlock+0x75/0x190 [ 6600.275475] ? _raw_spin_unlock+0x12/0x30 [ 6600.277441] do_sys_openat2+0x2b4/0x410 [ 6600.279588] do_sys_open+0x73/0xa0 [ 6600.281638] __x64_sys_openat+0x24/0x30 [ 6600.282774] do_syscall_64+0xc1/0x440 [ 6600.284496] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6600.287438] RIP: 0033:0x7f1a63983332 [ 6600.288797] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6600.300092] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6600.305194] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6600.307328] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6600.309632] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6600.312374] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6600.315158] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6602.721507] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4579s! [ 6602.731778] Showing busy workqueues and worker pools: [ 6602.738865] workqueue events: flags=0x0 [ 6602.742026] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6602.750103] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6602.757514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6602.763178] pending: sec_gc_main [ptlrpc] [ 6602.768852] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6602.776575] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6602.787991] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6602.795286] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6602.804209] workqueue events_long: flags=0x0 [ 6602.807928] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6602.813857] pending: lu_objects_flush [obdclass] [ 6602.815683] workqueue events_unbound: flags=0x2 [ 6602.817482] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6602.819961] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6602.824220] workqueue events_power_efficient: flags=0x80 [ 6602.826982] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6602.829609] pending: neigh_periodic_work, do_cache_clean [ 6602.831750] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6602.834386] pending: neigh_periodic_work, check_lifetime [ 6602.842525] workqueue mm_percpu_wq: flags=0x8 [ 6602.844198] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6602.847540] pending: vmstat_update [ 6602.848976] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6602.851726] pending: vmstat_update [ 6602.853395] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6602.859124] pending: vmstat_update [ 6602.860469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6602.864630] pending: vmstat_update [ 6602.867053] workqueue writeback: flags=0x4a [ 6602.868954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6602.872183] pending: wb_workfn, wb_workfn [ 6602.874096] workqueue xprtiod: flags=0xa [ 6602.875855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6602.878302] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6602.883040] workqueue ipv6_addrconf: flags=0x40008 [ 6602.885070] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6602.887498] pending: addrconf_verify_work [ 6602.889468] workqueue knbd0-recv: flags=0x1a [ 6602.891531] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6602.894327] in-flight: 647:recv_work [ 6602.896774] workqueue ptlrpc_pinger: flags=0x2 [ 6602.898782] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6602.901162] pending: ptlrpc_pinger_main [ptlrpc] [ 6602.903368] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6592s workers=2 idle: 652 [ 6608.056031] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 6608.059465] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6608.062459] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [ 6608.072438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6608.072514] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6608.072521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6608.072524] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6608.072537] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6608.072541] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6608.072545] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6608.072547] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6608.072550] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6608.072552] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6608.072554] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6608.072557] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6608.072560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6608.072562] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6608.072571] Call Trace: [ 6608.072576] [ 6608.072582] ? show_regs.cold.9+0x22/0x2f [ 6608.072591] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6608.072597] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6608.072601] ? watchdog+0x40/0x40 [ 6608.072605] ? hrtimer_interrupt+0x128/0x2c0 [ 6608.072609] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6608.072616] ? apic_timer_interrupt+0xf/0x20 [ 6608.072621] [ 6608.072624] ? native_safe_halt+0x2e/0x50 [ 6608.072629] kvm_wait+0xcd/0xe0 [ 6608.072636] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6608.072644] do_raw_spin_lock+0x177/0x190 [ 6608.072649] _raw_spin_lock+0x21/0x30 [ 6608.072654] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6608.082504] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6608.082571] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6608.082577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6608.082580] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6608.082593] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6608.082597] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6608.082601] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6608.082603] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6608.082606] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6608.082608] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6608.082610] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6608.082613] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6608.082616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6608.082618] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6608.082628] Call Trace: [ 6608.082632] [ 6608.082638] ? show_regs.cold.9+0x22/0x2f [ 6608.082646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6608.082654] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6608.082658] ? watchdog+0x40/0x40 [ 6608.082661] ? hrtimer_interrupt+0x128/0x2c0 [ 6608.082665] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6608.082672] ? apic_timer_interrupt+0xf/0x20 [ 6608.082676] [ 6608.082679] ? native_safe_halt+0x2e/0x50 [ 6608.082683] kvm_wait+0xcd/0xe0 [ 6608.082690] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6608.082698] do_raw_spin_lock+0x177/0x190 [ 6608.082703] _raw_spin_lock+0x21/0x30 [ 6608.082896] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6608.083206] ? do_raw_spin_unlock+0x75/0x190 [ 6608.083212] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6608.083407] ? do_wait_intr+0xf0/0xf0 [ 6608.083415] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6608.084393] mdc_close+0x53b/0xd70 [mdc] [ 6608.084429] lmv_close+0x18d/0x490 [lmv] [ 6608.084457] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6608.084662] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6608.084811] ll_md_close+0x274/0x800 [lustre] [ 6608.084982] ll_file_release+0x128/0xc50 [lustre] [ 6608.085131] __fput+0x10d/0x440 [ 6608.085140] ____fput+0x12/0x20 [ 6608.085145] task_work_run+0xc6/0x110 [ 6608.085152] exit_to_usermode_loop+0x1dd/0x1f0 [ 6608.085159] do_syscall_64+0x3d6/0x440 [ 6608.085164] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6608.085283] RIP: 0033:0x7f1a63983cd8 [ 6608.085291] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6608.085294] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6608.085298] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6608.085300] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6608.085302] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6608.085304] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6608.085306] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6608.115986] obd_mod_free+0x46/0x140 [ptlrpc] [ 6608.453542] mdc_commit_open+0x91/0xb0 [mdc] [ 6608.454971] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6608.456683] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6608.458614] after_reply+0xe2c/0x1780 [ptlrpc] [ 6608.460230] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6608.462032] ? __next_timer_interrupt+0x160/0x160 [ 6608.463431] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6608.464960] ? do_wait_intr+0xf0/0xf0 [ 6608.466028] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6608.467843] kthread+0x1d1/0x200 [ 6608.468743] ? set_kthread_struct+0x70/0x70 [ 6608.470031] ret_from_fork+0x1f/0x30 [ 6624.057196] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 6624.060842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6624.103031] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6624.109791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6624.115061] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6624.118185] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6624.128977] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6624.132773] RAX: 0000000000002022 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6624.136149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6624.140159] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6624.146613] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6624.150117] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6624.153815] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6624.157416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6624.160003] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6624.164011] Call Trace: [ 6624.165235] [ 6624.166285] ? show_regs.cold.9+0x22/0x2f [ 6624.168268] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6624.170981] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6624.173283] ? watchdog+0x40/0x40 [ 6624.174976] ? hrtimer_interrupt+0x128/0x2c0 [ 6624.177826] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6624.180455] ? apic_timer_interrupt+0xf/0x20 [ 6624.182922] [ 6624.183998] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6624.187053] do_raw_spin_lock+0x177/0x190 [ 6624.188853] _raw_spin_lock+0x21/0x30 [ 6624.190457] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6624.192280] ? do_raw_spin_unlock+0x75/0x190 [ 6624.193515] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6624.195801] ? do_wait_intr+0xf0/0xf0 [ 6624.197193] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6624.199419] mdc_close+0x53b/0xd70 [mdc] [ 6624.201176] lmv_close+0x18d/0x490 [lmv] [ 6624.202815] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6624.205048] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6624.206513] ll_md_close+0x274/0x800 [lustre] [ 6624.208219] ll_file_release+0x128/0xc50 [lustre] [ 6624.210005] __fput+0x10d/0x440 [ 6624.211160] ____fput+0x12/0x20 [ 6624.212634] task_work_run+0xc6/0x110 [ 6624.214192] exit_to_usermode_loop+0x1dd/0x1f0 [ 6624.215907] do_syscall_64+0x3d6/0x440 [ 6624.217289] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6624.219111] RIP: 0033:0x7f1a63983cd8 [ 6624.220107] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6624.225769] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6624.227894] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6624.229793] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6624.232240] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6624.234159] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6624.236273] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6628.053627] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6628.060724] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6628.090479] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6628.102042] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6628.109367] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6628.111531] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6628.123190] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6628.127233] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6628.132114] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6628.135556] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6628.137808] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6628.142309] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6628.145173] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6628.152333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6628.157168] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6628.166420] Call Trace: [ 6628.169109] [ 6628.170356] ? show_regs.cold.9+0x22/0x2f [ 6628.172438] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6628.174898] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6628.176565] ? watchdog+0x40/0x40 [ 6628.179484] ? hrtimer_interrupt+0x128/0x2c0 [ 6628.182395] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6628.185431] ? apic_timer_interrupt+0xf/0x20 [ 6628.187806] [ 6628.189042] ? native_safe_halt+0x2e/0x50 [ 6628.192893] kvm_wait+0xcd/0xe0 [ 6628.194145] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6628.197211] do_raw_spin_lock+0x177/0x190 [ 6628.203951] _raw_spin_lock+0x21/0x30 [ 6628.207206] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6628.214382] ? do_raw_spin_unlock+0x75/0x190 [ 6628.218006] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6628.226845] ? do_wait_intr+0xf0/0xf0 [ 6628.229189] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6628.234955] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6628.240471] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6628.244188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6628.247520] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6628.251091] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6628.254132] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6628.256024] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6628.258930] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6628.264451] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6628.268460] lmv_intent_open+0x338/0x1150 [lmv] [ 6628.271253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6628.274273] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6628.277238] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6628.279449] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6628.282172] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6628.285210] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6628.288354] ? do_raw_spin_unlock+0x75/0x190 [ 6628.297064] ? _raw_spin_unlock+0x12/0x30 [ 6628.299395] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6628.303785] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6628.307954] ll_atomic_open+0x36a/0xd70 [lustre] [ 6628.311268] lookup_open+0x356/0xb40 [ 6628.317072] do_last+0x313/0xfc0 [ 6628.318489] ? nd_jump_root+0xe5/0x160 [ 6628.324033] ? path_init+0x437/0x520 [ 6628.325376] path_openat+0xf7/0x500 [ 6628.326613] ? do_raw_spin_unlock+0x75/0x190 [ 6628.339857] do_filp_open+0x99/0x140 [ 6628.345919] ? getname_flags+0x6e/0x330 [ 6628.347383] ? __check_object_size+0xff/0x256 [ 6628.354334] ? do_raw_spin_unlock+0x75/0x190 [ 6628.357337] ? _raw_spin_unlock+0x12/0x30 [ 6628.361164] do_sys_openat2+0x2b4/0x410 [ 6628.368066] do_sys_open+0x73/0xa0 [ 6628.372248] __x64_sys_openat+0x24/0x30 [ 6628.379907] do_syscall_64+0xc1/0x440 [ 6628.383245] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6628.389200] RIP: 0033:0x7f1a63983332 [ 6628.391298] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6628.415037] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6628.424496] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6628.431927] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6628.440289] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6628.447813] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6628.454037] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6632.929049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4609s! [ 6632.932241] Showing busy workqueues and worker pools: [ 6632.939034] workqueue events: flags=0x0 [ 6632.940535] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6632.949919] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6632.961321] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6632.973151] pending: sec_gc_main [ptlrpc] [ 6632.978338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6632.991026] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6633.005774] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6633.008457] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6633.020916] workqueue events_long: flags=0x0 [ 6633.022498] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6633.028921] pending: lu_objects_flush [obdclass] [ 6633.031955] workqueue events_unbound: flags=0x2 [ 6633.039414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6633.045568] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6633.055480] workqueue events_power_efficient: flags=0x80 [ 6633.063246] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6633.068382] pending: neigh_periodic_work, do_cache_clean [ 6633.075510] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6633.080936] pending: neigh_periodic_work, check_lifetime [ 6633.084132] workqueue mm_percpu_wq: flags=0x8 [ 6633.086122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6633.089422] pending: vmstat_update [ 6633.091466] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6633.094550] pending: vmstat_update [ 6633.098193] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6633.103204] pending: vmstat_update [ 6633.105982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6633.113443] pending: vmstat_update [ 6633.115435] workqueue writeback: flags=0x4a [ 6633.118479] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6633.122788] pending: wb_workfn, wb_workfn [ 6633.124522] workqueue xprtiod: flags=0xa [ 6633.130818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6633.134661] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6633.143148] workqueue ipv6_addrconf: flags=0x40008 [ 6633.146181] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6633.151338] pending: addrconf_verify_work [ 6633.154851] workqueue knbd0-recv: flags=0x1a [ 6633.159220] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6633.166379] in-flight: 647:recv_work [ 6633.170567] workqueue ptlrpc_pinger: flags=0x2 [ 6633.173721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6633.177673] pending: ptlrpc_pinger_main [ptlrpc] [ 6633.180469] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6623s workers=2 idle: 652 [ 6636.056576] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 6636.058159] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6636.059190] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [ 6636.062635] Modules linked in: lustre(O) [ 6636.066606] lquota(O) lfsck(O) obdecho(O) [ 6636.068514] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [ 6636.071157] mgc(O) mdc(O) lov(O) osc(O) [ 6636.076422] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 6636.078678] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [ 6636.085103] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6636.085152] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6636.085160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6636.085162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6636.085176] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6636.085180] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6636.085184] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6636.085186] RDX: 0000000000005f87 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6636.085189] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6636.085191] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6636.085193] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6636.085197] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6636.085200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6636.085203] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6636.085211] Call Trace: [ 6636.085217] [ 6636.085223] ? show_regs.cold.9+0x22/0x2f [ 6636.085231] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6636.085237] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6636.085241] ? watchdog+0x40/0x40 [ 6636.085245] ? hrtimer_interrupt+0x128/0x2c0 [ 6636.085249] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6636.085263] ? apic_timer_interrupt+0xf/0x20 [ 6636.085268] [ 6636.085273] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6636.089538] libcfs(O) rpcsec_gss_krb5 [ 6636.106514] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6636.111776] auth_rpcgss [ 6636.114831] do_raw_spin_lock+0x177/0x190 [ 6636.118179] nfsv4 dns_resolver [ 6636.129109] _raw_spin_lock+0x21/0x30 [ 6636.132905] intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6636.132993] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6636.133000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6636.133003] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6636.133086] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6636.133092] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6636.133096] RAX: 0000000000007247 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6636.133099] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6636.133101] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6636.133103] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6636.133106] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6636.133110] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6636.133113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6636.133116] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6636.133124] Call Trace: [ 6636.133129] [ 6636.133134] ? show_regs.cold.9+0x22/0x2f [ 6636.133143] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6636.133150] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6636.133154] ? watchdog+0x40/0x40 [ 6636.133159] ? hrtimer_interrupt+0x128/0x2c0 [ 6636.133164] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6636.133172] ? apic_timer_interrupt+0xf/0x20 [ 6636.133177] [ 6636.133179] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6636.133185] do_raw_spin_lock+0x177/0x190 [ 6636.133189] _raw_spin_lock+0x21/0x30 [ 6636.133194] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6636.136711] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6636.137119] obd_mod_free+0x46/0x140 [ptlrpc] [ 6636.137336] mdc_commit_open+0x91/0xb0 [mdc] [ 6636.137376] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6636.137597] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6636.137937] after_reply+0xe2c/0x1780 [ptlrpc] [ 6636.138250] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6636.138456] ? __next_timer_interrupt+0x160/0x160 [ 6636.138467] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6636.138775] ? do_wait_intr+0xf0/0xf0 [ 6636.138785] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6636.139112] kthread+0x1d1/0x200 [ 6636.139119] ? set_kthread_struct+0x70/0x70 [ 6636.139122] ret_from_fork+0x1f/0x30 [ 6636.316001] ? do_raw_spin_unlock+0x75/0x190 [ 6636.317699] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6636.319531] ? do_wait_intr+0xf0/0xf0 [ 6636.320911] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6636.322822] mdc_close+0x53b/0xd70 [mdc] [ 6636.324156] lmv_close+0x18d/0x490 [lmv] [ 6636.325685] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6636.327641] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6636.329310] ll_md_close+0x274/0x800 [lustre] [ 6636.330835] ll_file_release+0x128/0xc50 [lustre] [ 6636.332530] __fput+0x10d/0x440 [ 6636.333535] ____fput+0x12/0x20 [ 6636.334632] task_work_run+0xc6/0x110 [ 6636.335794] exit_to_usermode_loop+0x1dd/0x1f0 [ 6636.337252] do_syscall_64+0x3d6/0x440 [ 6636.338352] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6636.339901] RIP: 0033:0x7f1a63983cd8 [ 6636.341140] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6636.347006] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6636.349777] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6636.351953] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6636.354241] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6636.356673] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6636.358865] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6652.057030] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 6652.061918] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6652.103246] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6652.113266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6652.128438] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6652.130187] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6652.150102] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6652.162238] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6652.172908] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6652.176828] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6652.182130] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6652.186892] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6652.196045] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6652.200955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6652.208619] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6652.212976] Call Trace: [ 6652.214035] [ 6652.214977] ? show_regs.cold.9+0x22/0x2f [ 6652.216904] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6652.218990] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6652.221258] ? watchdog+0x40/0x40 [ 6652.222958] ? hrtimer_interrupt+0x128/0x2c0 [ 6652.227349] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6652.231914] ? apic_timer_interrupt+0xf/0x20 [ 6652.234474] [ 6652.237414] ? native_safe_halt+0x2e/0x50 [ 6652.240298] kvm_wait+0xcd/0xe0 [ 6652.242226] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6652.245334] do_raw_spin_lock+0x177/0x190 [ 6652.250702] _raw_spin_lock+0x21/0x30 [ 6652.252924] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6652.255585] ? do_raw_spin_unlock+0x75/0x190 [ 6652.263012] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6652.264564] ? do_wait_intr+0xf0/0xf0 [ 6652.266361] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6652.269865] mdc_close+0x53b/0xd70 [mdc] [ 6652.271178] lmv_close+0x18d/0x490 [lmv] [ 6652.274151] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6652.277657] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6652.283807] ll_md_close+0x274/0x800 [lustre] [ 6652.286785] ll_file_release+0x128/0xc50 [lustre] [ 6652.289081] __fput+0x10d/0x440 [ 6652.293870] ____fput+0x12/0x20 [ 6652.295048] task_work_run+0xc6/0x110 [ 6652.297549] exit_to_usermode_loop+0x1dd/0x1f0 [ 6652.300879] do_syscall_64+0x3d6/0x440 [ 6652.302837] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6652.306138] RIP: 0033:0x7f1a63983cd8 [ 6652.308950] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6652.327063] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6652.334004] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6652.337971] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6652.343847] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6652.348666] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6652.352798] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6660.057575] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6660.066726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6660.107746] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6660.119651] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6660.123872] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6660.127802] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6660.146533] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6660.151029] RAX: 0000000000006430 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6660.157098] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6660.160416] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6660.167777] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6660.175559] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6660.180424] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6660.190193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6660.193609] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6660.197858] Call Trace: [ 6660.202312] [ 6660.203322] ? show_regs.cold.9+0x22/0x2f [ 6660.205132] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6660.209902] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6660.214407] ? watchdog+0x40/0x40 [ 6660.216127] ? hrtimer_interrupt+0x128/0x2c0 [ 6660.218772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6660.221530] ? apic_timer_interrupt+0xf/0x20 [ 6660.225890] [ 6660.227908] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6660.236107] do_raw_spin_lock+0x177/0x190 [ 6660.253515] _raw_spin_lock+0x21/0x30 [ 6660.254638] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6660.261860] ? do_raw_spin_unlock+0x75/0x190 [ 6660.263291] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6660.268617] ? do_wait_intr+0xf0/0xf0 [ 6660.273460] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6660.275566] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6660.284776] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6660.288062] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6660.293528] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6660.305609] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6660.316586] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6660.321698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6660.326516] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6660.329831] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6660.335455] lmv_intent_open+0x338/0x1150 [lmv] [ 6660.338146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6660.342810] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6660.347926] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6660.352977] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6660.354737] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6660.365493] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6660.370525] ? do_raw_spin_unlock+0x75/0x190 [ 6660.374662] ? _raw_spin_unlock+0x12/0x30 [ 6660.378715] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6660.381880] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6660.388511] ll_atomic_open+0x36a/0xd70 [lustre] [ 6660.395078] lookup_open+0x356/0xb40 [ 6660.399699] do_last+0x313/0xfc0 [ 6660.403921] ? nd_jump_root+0xe5/0x160 [ 6660.405269] ? path_init+0x437/0x520 [ 6660.410124] path_openat+0xf7/0x500 [ 6660.412812] ? do_raw_spin_unlock+0x75/0x190 [ 6660.414762] do_filp_open+0x99/0x140 [ 6660.422638] ? getname_flags+0x6e/0x330 [ 6660.426794] ? __check_object_size+0xff/0x256 [ 6660.432739] ? do_raw_spin_unlock+0x75/0x190 [ 6660.435618] ? _raw_spin_unlock+0x12/0x30 [ 6660.436847] do_sys_openat2+0x2b4/0x410 [ 6660.440775] do_sys_open+0x73/0xa0 [ 6660.442983] __x64_sys_openat+0x24/0x30 [ 6660.444105] do_syscall_64+0xc1/0x440 [ 6660.446505] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6660.448531] RIP: 0033:0x7f1a63983332 [ 6660.450757] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6660.466937] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6660.475745] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6660.479587] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6660.485516] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6660.491827] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6660.499506] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6663.653578] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4640s! [ 6663.660327] Showing busy workqueues and worker pools: [ 6663.662669] workqueue events: flags=0x0 [ 6663.664369] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6663.667151] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6663.671147] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6663.674767] pending: sec_gc_main [ptlrpc] [ 6663.677356] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6663.680494] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6663.684856] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6663.690392] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6663.696828] workqueue events_long: flags=0x0 [ 6663.704800] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6663.713979] pending: lu_objects_flush [obdclass] [ 6663.721071] workqueue events_unbound: flags=0x2 [ 6663.724011] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6663.727938] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6663.733321] workqueue events_power_efficient: flags=0x80 [ 6663.735778] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6663.738883] pending: neigh_periodic_work, do_cache_clean [ 6663.741696] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6663.745031] pending: neigh_periodic_work, check_lifetime [ 6663.748151] workqueue mm_percpu_wq: flags=0x8 [ 6663.750535] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6663.753723] pending: vmstat_update [ 6663.755698] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6663.758589] pending: vmstat_update [ 6663.760595] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6663.763739] pending: vmstat_update [ 6663.765273] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6663.769151] pending: vmstat_update [ 6663.770884] workqueue writeback: flags=0x4a [ 6663.773187] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6663.775871] pending: wb_workfn, wb_workfn [ 6663.778369] workqueue xprtiod: flags=0xa [ 6663.780131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6663.782710] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6663.787944] workqueue ipv6_addrconf: flags=0x40008 [ 6663.791172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6663.794962] pending: addrconf_verify_work [ 6663.797204] workqueue knbd0-recv: flags=0x1a [ 6663.799484] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6663.802758] in-flight: 647:recv_work [ 6663.804814] workqueue ptlrpc_pinger: flags=0x2 [ 6663.807207] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6663.809644] pending: ptlrpc_pinger_main [ptlrpc] [ 6663.812472] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6653s workers=2 idle: 652 [ 6664.059919] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6664.063006] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6664.063364] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6664.081302] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6664.081314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6664.081317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6664.081330] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6664.081334] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6664.081340] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6664.081343] RDX: 0000000000003649 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6664.081345] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6664.081347] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6664.081349] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6664.081353] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6664.081356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6664.081358] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6664.081366] Call Trace: [ 6664.081371] [ 6664.081377] ? show_regs.cold.9+0x22/0x2f [ 6664.081386] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6664.081392] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6664.081396] ? watchdog+0x40/0x40 [ 6664.081400] ? hrtimer_interrupt+0x128/0x2c0 [ 6664.081404] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6664.081412] ? apic_timer_interrupt+0xf/0x20 [ 6664.081417] [ 6664.081420] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6664.081424] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6664.081429] do_raw_spin_lock+0x177/0x190 [ 6664.081434] _raw_spin_lock+0x21/0x30 [ 6664.081440] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6664.081762] obd_mod_free+0x46/0x140 [ptlrpc] [ 6664.081972] mdc_commit_open+0x91/0xb0 [mdc] [ 6664.082015] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6664.082243] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6664.082937] after_reply+0xe2c/0x1780 [ptlrpc] [ 6664.083193] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6664.083475] ? __next_timer_interrupt+0x160/0x160 [ 6664.083485] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6664.084469] ? do_wait_intr+0xf0/0xf0 [ 6664.084479] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6664.084682] kthread+0x1d1/0x200 [ 6664.084687] ? set_kthread_struct+0x70/0x70 [ 6664.084691] ret_from_fork+0x1f/0x30 [ 6664.328592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6664.346561] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6664.353903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6664.357402] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6664.359741] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6664.374956] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6664.387013] RAX: 0000000000005f77 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6664.393753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6664.400741] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6664.406906] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6664.415509] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6664.423867] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6664.427197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6664.430016] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6664.433357] Call Trace: [ 6664.434112] [ 6664.434937] ? show_regs.cold.9+0x22/0x2f [ 6664.436818] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6664.438418] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6664.439753] ? watchdog+0x40/0x40 [ 6664.440704] ? hrtimer_interrupt+0x128/0x2c0 [ 6664.442008] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6664.443607] ? apic_timer_interrupt+0xf/0x20 [ 6664.444749] [ 6664.445448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6664.447022] do_raw_spin_lock+0x177/0x190 [ 6664.448627] _raw_spin_lock+0x21/0x30 [ 6664.449982] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6664.455653] ? do_raw_spin_unlock+0x75/0x190 [ 6664.459602] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6664.463591] ? do_wait_intr+0xf0/0xf0 [ 6664.468364] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6664.470713] mdc_close+0x53b/0xd70 [mdc] [ 6664.473121] lmv_close+0x18d/0x490 [lmv] [ 6664.478385] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6664.480431] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6664.482471] ll_md_close+0x274/0x800 [lustre] [ 6664.484677] ll_file_release+0x128/0xc50 [lustre] [ 6664.487199] __fput+0x10d/0x440 [ 6664.488634] ____fput+0x12/0x20 [ 6664.490640] task_work_run+0xc6/0x110 [ 6664.503026] exit_to_usermode_loop+0x1dd/0x1f0 [ 6664.511608] do_syscall_64+0x3d6/0x440 [ 6664.515158] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6664.519606] RIP: 0033:0x7f1a63983cd8 [ 6664.523021] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6664.536526] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6664.542566] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6664.545467] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6664.548061] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6664.551116] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6664.554068] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6680.057475] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 6680.063318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6680.119342] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6680.138599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6680.151185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6680.158197] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6680.175852] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6680.185542] RAX: 0000000000007e97 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6680.191108] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6680.196209] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6680.199987] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6680.203526] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6680.206956] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6680.210874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6680.214064] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6680.217737] Call Trace: [ 6680.218685] [ 6680.222463] ? show_regs.cold.9+0x22/0x2f [ 6680.223947] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6680.231586] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6680.236621] ? watchdog+0x40/0x40 [ 6680.238969] ? hrtimer_interrupt+0x128/0x2c0 [ 6680.241576] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6680.245688] ? apic_timer_interrupt+0xf/0x20 [ 6680.247839] [ 6680.249657] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6680.256779] do_raw_spin_lock+0x177/0x190 [ 6680.261536] _raw_spin_lock+0x21/0x30 [ 6680.263980] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6680.267114] ? do_raw_spin_unlock+0x75/0x190 [ 6680.269373] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6680.271970] ? do_wait_intr+0xf0/0xf0 [ 6680.273572] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6680.279422] mdc_close+0x53b/0xd70 [mdc] [ 6680.280946] lmv_close+0x18d/0x490 [lmv] [ 6680.283970] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6680.289923] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6680.295779] ll_md_close+0x274/0x800 [lustre] [ 6680.298662] ll_file_release+0x128/0xc50 [lustre] [ 6680.301157] __fput+0x10d/0x440 [ 6680.302666] ____fput+0x12/0x20 [ 6680.304203] task_work_run+0xc6/0x110 [ 6680.306078] exit_to_usermode_loop+0x1dd/0x1f0 [ 6680.308220] do_syscall_64+0x3d6/0x440 [ 6680.309397] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6680.311429] RIP: 0033:0x7f1a63983cd8 [ 6680.313798] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6680.323444] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6680.332695] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6680.337252] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6680.345000] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6680.349472] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6680.360737] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6688.053023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6688.057214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6688.086767] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6688.091959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6688.096752] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6688.099401] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6688.107517] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6688.111645] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6688.115583] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6688.121379] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6688.127816] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6688.134409] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6688.137777] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6688.140663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6688.142447] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6688.144462] Call Trace: [ 6688.145448] [ 6688.146030] ? show_regs.cold.9+0x22/0x2f [ 6688.147317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6688.148534] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6688.150426] ? watchdog+0x40/0x40 [ 6688.151406] ? hrtimer_interrupt+0x128/0x2c0 [ 6688.153337] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6688.155639] ? apic_timer_interrupt+0xf/0x20 [ 6688.157667] [ 6688.159283] ? native_safe_halt+0x2e/0x50 [ 6688.161294] kvm_wait+0xcd/0xe0 [ 6688.162715] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6688.166166] do_raw_spin_lock+0x177/0x190 [ 6688.167587] _raw_spin_lock+0x21/0x30 [ 6688.169639] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6688.172488] ? do_raw_spin_unlock+0x75/0x190 [ 6688.174880] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6688.180643] ? do_wait_intr+0xf0/0xf0 [ 6688.182598] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6688.184915] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6688.187519] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6688.190430] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6688.192899] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6688.196255] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6688.202442] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6688.205796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6688.214751] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6688.221270] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6688.224150] lmv_intent_open+0x338/0x1150 [lmv] [ 6688.226738] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6688.230125] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6688.232976] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6688.236019] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6688.238730] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6688.241677] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6688.244205] ? do_raw_spin_unlock+0x75/0x190 [ 6688.247243] ? _raw_spin_unlock+0x12/0x30 [ 6688.249307] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6688.252218] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6688.253915] ll_atomic_open+0x36a/0xd70 [lustre] [ 6688.256611] lookup_open+0x356/0xb40 [ 6688.258649] do_last+0x313/0xfc0 [ 6688.260413] ? nd_jump_root+0xe5/0x160 [ 6688.262412] ? path_init+0x437/0x520 [ 6688.264457] path_openat+0xf7/0x500 [ 6688.266274] ? do_raw_spin_unlock+0x75/0x190 [ 6688.267630] do_filp_open+0x99/0x140 [ 6688.268707] ? getname_flags+0x6e/0x330 [ 6688.270645] ? __check_object_size+0xff/0x256 [ 6688.272759] ? do_raw_spin_unlock+0x75/0x190 [ 6688.274874] ? _raw_spin_unlock+0x12/0x30 [ 6688.276650] do_sys_openat2+0x2b4/0x410 [ 6688.278309] do_sys_open+0x73/0xa0 [ 6688.280384] __x64_sys_openat+0x24/0x30 [ 6688.282299] do_syscall_64+0xc1/0x440 [ 6688.285343] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6688.288195] RIP: 0033:0x7f1a63983332 [ 6688.289740] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6688.299705] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6688.307458] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6688.312545] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6688.316664] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6688.320734] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6688.324438] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6692.056168] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6692.060184] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6692.064752] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6692.090750] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6692.090762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6692.090765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6692.090776] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6692.090779] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6692.090784] RAX: 0000000000001e92 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6692.090786] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6692.090788] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6692.090789] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6692.090791] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6692.090794] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6692.101001] Modules linked in: [ 6692.104492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6692.104501] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6692.104510] Call Trace: [ 6692.104517] [ 6692.104523] ? show_regs.cold.9+0x22/0x2f [ 6692.104535] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6692.104542] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6692.104546] ? watchdog+0x40/0x40 [ 6692.104551] ? hrtimer_interrupt+0x128/0x2c0 [ 6692.104556] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6692.104564] ? apic_timer_interrupt+0xf/0x20 [ 6692.104569] [ 6692.104572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6692.104588] do_raw_spin_lock+0x177/0x190 [ 6692.104592] _raw_spin_lock+0x21/0x30 [ 6692.104597] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6692.105007] ? do_raw_spin_unlock+0x75/0x190 [ 6692.105014] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6692.105228] ? do_wait_intr+0xf0/0xf0 [ 6692.105237] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6692.105426] mdc_close+0x53b/0xd70 [mdc] [ 6692.105466] lmv_close+0x18d/0x490 [lmv] [ 6692.105498] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6692.105613] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6692.105694] ll_md_close+0x274/0x800 [lustre] [ 6692.105915] ll_file_release+0x128/0xc50 [lustre] [ 6692.105998] __fput+0x10d/0x440 [ 6692.106007] ____fput+0x12/0x20 [ 6692.106012] task_work_run+0xc6/0x110 [ 6692.106031] exit_to_usermode_loop+0x1dd/0x1f0 [ 6692.106038] do_syscall_64+0x3d6/0x440 [ 6692.106043] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6692.106050] RIP: 0033:0x7f1a63983cd8 [ 6692.106113] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6692.106117] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6692.106122] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6692.106124] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6692.106127] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6692.106129] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6692.106131] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6692.411522] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6692.454648] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6692.462133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6692.470763] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6692.476601] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6692.489964] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6692.506848] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6692.514525] RDX: 0000000000007c1b RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6692.527849] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6692.536888] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6692.541484] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6692.551853] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6692.557974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6692.560875] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6692.566078] Call Trace: [ 6692.567005] [ 6692.572247] ? show_regs.cold.9+0x22/0x2f [ 6692.573549] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6692.574771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6692.576547] ? watchdog+0x40/0x40 [ 6692.579986] ? hrtimer_interrupt+0x128/0x2c0 [ 6692.585252] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6692.586934] ? apic_timer_interrupt+0xf/0x20 [ 6692.600476] [ 6692.607216] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6692.609829] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6692.611877] do_raw_spin_lock+0x177/0x190 [ 6692.618535] _raw_spin_lock+0x21/0x30 [ 6692.620772] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6692.625399] obd_mod_free+0x46/0x140 [ptlrpc] [ 6692.630227] mdc_commit_open+0x91/0xb0 [mdc] [ 6692.634967] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6692.639836] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6692.642528] after_reply+0xe2c/0x1780 [ptlrpc] [ 6692.648561] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6692.652784] ? __next_timer_interrupt+0x160/0x160 [ 6692.656206] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6692.657371] ? do_wait_intr+0xf0/0xf0 [ 6692.658714] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6692.664787] kthread+0x1d1/0x200 [ 6692.666072] ? set_kthread_struct+0x70/0x70 [ 6692.667953] ret_from_fork+0x1f/0x30 [ 6693.856541] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4670s! [ 6693.866719] Showing busy workqueues and worker pools: [ 6693.877493] workqueue events: flags=0x0 [ 6693.885501] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6693.893621] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6693.897243] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6693.903241] pending: sec_gc_main [ptlrpc] [ 6693.907421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6693.913299] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6693.921533] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6693.925854] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6693.931055] workqueue events_long: flags=0x0 [ 6693.932823] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6693.936062] pending: lu_objects_flush [obdclass] [ 6693.938686] workqueue events_unbound: flags=0x2 [ 6693.940438] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6693.942987] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6693.947603] workqueue events_power_efficient: flags=0x80 [ 6693.951622] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6693.955165] pending: neigh_periodic_work, do_cache_clean [ 6693.957674] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6693.960553] pending: neigh_periodic_work, check_lifetime [ 6693.963919] workqueue mm_percpu_wq: flags=0x8 [ 6693.972642] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6693.978880] pending: vmstat_update [ 6693.982628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6693.988228] pending: vmstat_update [ 6693.989724] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6693.993588] pending: vmstat_update [ 6693.995816] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6693.999107] pending: vmstat_update [ 6694.000653] workqueue writeback: flags=0x4a [ 6694.009070] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6694.011347] pending: wb_workfn, wb_workfn [ 6694.018841] workqueue xprtiod: flags=0xa [ 6694.024218] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6694.027073] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6694.032549] workqueue ipv6_addrconf: flags=0x40008 [ 6694.035568] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6694.042957] pending: addrconf_verify_work [ 6694.048417] workqueue knbd0-recv: flags=0x1a [ 6694.051566] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6694.055406] in-flight: 647:recv_work [ 6694.057493] workqueue ptlrpc_pinger: flags=0x2 [ 6694.059619] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6694.063208] pending: ptlrpc_pinger_main [ptlrpc] [ 6694.066205] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6684s workers=2 idle: 652 [ 6708.059395] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6708.070026] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6708.118836] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6708.138798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6708.148227] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6708.154661] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6708.165174] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6708.169546] RAX: 0000000000007e8f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6708.173772] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6708.177542] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6708.181049] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6708.184851] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6708.188678] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6708.202493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6708.209105] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6708.212761] Call Trace: [ 6708.214654] [ 6708.215546] ? show_regs.cold.9+0x22/0x2f [ 6708.222361] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6708.224823] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6708.232480] ? watchdog+0x40/0x40 [ 6708.237439] ? hrtimer_interrupt+0x128/0x2c0 [ 6708.243535] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6708.249505] ? apic_timer_interrupt+0xf/0x20 [ 6708.253238] [ 6708.254537] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6708.259148] do_raw_spin_lock+0x177/0x190 [ 6708.263826] _raw_spin_lock+0x21/0x30 [ 6708.267480] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6708.274395] ? do_raw_spin_unlock+0x75/0x190 [ 6708.279761] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6708.285434] ? do_wait_intr+0xf0/0xf0 [ 6708.287080] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6708.296624] mdc_close+0x53b/0xd70 [mdc] [ 6708.302177] lmv_close+0x18d/0x490 [lmv] [ 6708.307999] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6708.315460] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6708.324095] ll_md_close+0x274/0x800 [lustre] [ 6708.327518] ll_file_release+0x128/0xc50 [lustre] [ 6708.336960] __fput+0x10d/0x440 [ 6708.345693] ____fput+0x12/0x20 [ 6708.347107] task_work_run+0xc6/0x110 [ 6708.354735] exit_to_usermode_loop+0x1dd/0x1f0 [ 6708.360767] do_syscall_64+0x3d6/0x440 [ 6708.367362] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6708.370260] RIP: 0033:0x7f1a63983cd8 [ 6708.374848] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6708.393509] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6708.403799] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6708.411446] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6708.421031] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6708.427087] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6708.431536] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6720.053017] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6720.056115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6720.058102] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6720.065677] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6720.065693] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6720.066593] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6720.066603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6720.066607] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6720.066621] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6720.066625] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6720.066630] RAX: 0000000000002f86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6720.066632] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6720.066635] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6720.066637] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6720.066639] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6720.066642] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6720.066645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6720.066647] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6720.066656] Call Trace: [ 6720.066662] [ 6720.066668] ? show_regs.cold.9+0x22/0x2f [ 6720.066679] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6720.066685] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6720.066691] ? watchdog+0x40/0x40 [ 6720.066695] ? hrtimer_interrupt+0x128/0x2c0 [ 6720.066700] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6720.066708] ? apic_timer_interrupt+0xf/0x20 [ 6720.066713] [ 6720.066717] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6720.066723] do_raw_spin_lock+0x177/0x190 [ 6720.066729] _raw_spin_lock+0x21/0x30 [ 6720.066734] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6720.067149] ? do_raw_spin_unlock+0x75/0x190 [ 6720.067155] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6720.067352] ? do_wait_intr+0xf0/0xf0 [ 6720.067360] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6720.068676] mdc_close+0x53b/0xd70 [mdc] [ 6720.068716] lmv_close+0x18d/0x490 [lmv] [ 6720.068756] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6720.068880] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6720.068954] ll_md_close+0x274/0x800 [lustre] [ 6720.069030] ll_file_release+0x128/0xc50 [lustre] [ 6720.069105] __fput+0x10d/0x440 [ 6720.069112] ____fput+0x12/0x20 [ 6720.069116] task_work_run+0xc6/0x110 [ 6720.069123] exit_to_usermode_loop+0x1dd/0x1f0 [ 6720.069129] do_syscall_64+0x3d6/0x440 [ 6720.069132] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6720.069140] RIP: 0033:0x7f1a63983cd8 [ 6720.069147] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6720.069150] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6720.069155] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6720.069157] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6720.069159] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6720.069162] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6720.069164] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6720.091092] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6720.091106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6720.091109] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6720.091122] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6720.091127] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6720.091133] RAX: 0000000000004484 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6720.091135] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6720.091138] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6720.091141] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6720.091143] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6720.091147] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6720.091150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6720.091153] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6720.091162] Call Trace: [ 6720.091167] [ 6720.091172] ? show_regs.cold.9+0x22/0x2f [ 6720.091181] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6720.091188] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6720.091193] ? watchdog+0x40/0x40 [ 6720.091197] ? hrtimer_interrupt+0x128/0x2c0 [ 6720.091202] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6720.091413] ? apic_timer_interrupt+0xf/0x20 [ 6720.091420] [ 6720.091424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6720.091431] do_raw_spin_lock+0x177/0x190 [ 6720.091436] _raw_spin_lock+0x21/0x30 [ 6720.091441] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6720.091756] ? do_raw_spin_unlock+0x75/0x190 [ 6720.091763] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6720.091973] ? do_wait_intr+0xf0/0xf0 [ 6720.092044] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6720.092449] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6720.099730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6720.100262] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6720.100270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6720.100274] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6720.100289] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6720.100294] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6720.100298] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6720.100301] RDX: 0000000000003e87 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6720.100303] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6720.100305] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6720.100308] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6720.100311] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6720.100314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6720.100316] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6720.100325] Call Trace: [ 6720.100330] [ 6720.100336] ? show_regs.cold.9+0x22/0x2f [ 6720.100345] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6720.100351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6720.100355] ? watchdog+0x40/0x40 [ 6720.100359] ? hrtimer_interrupt+0x128/0x2c0 [ 6720.100363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6720.100371] ? apic_timer_interrupt+0xf/0x20 [ 6720.100376] [ 6720.100379] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6720.100385] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6720.100389] do_raw_spin_lock+0x177/0x190 [ 6720.100394] _raw_spin_lock+0x21/0x30 [ 6720.100399] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6720.100669] obd_mod_free+0x46/0x140 [ptlrpc] [ 6720.106230] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6720.106625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6720.106910] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6720.106957] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6720.106988] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6720.107018] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6720.107122] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6720.107377] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6720.107409] lmv_intent_open+0x338/0x1150 [lmv] [ 6720.107440] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6720.107535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6720.107627] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6720.107862] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6720.107944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6720.108023] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6720.108102] ? do_raw_spin_unlock+0x75/0x190 [ 6720.108109] ? _raw_spin_unlock+0x12/0x30 [ 6720.108115] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6720.108144] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6720.108160] ll_atomic_open+0x36a/0xd70 [lustre] [ 6720.108249] lookup_open+0x356/0xb40 [ 6720.108258] do_last+0x313/0xfc0 [ 6720.108263] ? nd_jump_root+0xe5/0x160 [ 6720.108268] ? path_init+0x437/0x520 [ 6720.108273] path_openat+0xf7/0x500 [ 6720.108277] ? do_raw_spin_unlock+0x75/0x190 [ 6720.108282] do_filp_open+0x99/0x140 [ 6720.108288] ? getname_flags+0x6e/0x330 [ 6720.108292] ? __check_object_size+0xff/0x256 [ 6720.108299] ? do_raw_spin_unlock+0x75/0x190 [ 6720.108303] ? _raw_spin_unlock+0x12/0x30 [ 6720.108308] do_sys_openat2+0x2b4/0x410 [ 6720.108312] do_sys_open+0x73/0xa0 [ 6720.108317] __x64_sys_openat+0x24/0x30 [ 6720.108381] do_syscall_64+0xc1/0x440 [ 6720.108390] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6720.108397] RIP: 0033:0x7f1a63983332 [ 6720.108405] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6720.108409] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6720.108413] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6720.108416] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6720.108418] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6720.108419] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6720.108422] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6720.586029] mdc_commit_open+0x91/0xb0 [mdc] [ 6720.587696] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6720.589853] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6720.591614] after_reply+0xe2c/0x1780 [ptlrpc] [ 6720.593317] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6720.595345] ? __next_timer_interrupt+0x160/0x160 [ 6720.597256] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6720.599188] ? do_wait_intr+0xf0/0xf0 [ 6720.600710] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6720.603063] kthread+0x1d1/0x200 [ 6720.604419] ? set_kthread_struct+0x70/0x70 [ 6720.606097] ret_from_fork+0x1f/0x30 [ 6724.576848] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4701s! [ 6724.591324] Showing busy workqueues and worker pools: [ 6724.596432] workqueue events: flags=0x0 [ 6724.599018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6724.602186] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6724.606044] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6724.609244] pending: sec_gc_main [ptlrpc] [ 6724.612595] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6724.615471] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6724.624399] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6724.631004] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6724.636683] workqueue events_long: flags=0x0 [ 6724.639674] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6724.643896] pending: lu_objects_flush [obdclass] [ 6724.646938] workqueue events_unbound: flags=0x2 [ 6724.649133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6724.652627] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6724.662153] workqueue events_power_efficient: flags=0x80 [ 6724.669636] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6724.673883] pending: neigh_periodic_work, do_cache_clean [ 6724.677395] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6724.681614] pending: neigh_periodic_work, check_lifetime [ 6724.684867] workqueue mm_percpu_wq: flags=0x8 [ 6724.687203] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6724.691175] pending: vmstat_update [ 6724.694045] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6724.699185] pending: vmstat_update [ 6724.702047] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6724.705589] pending: vmstat_update [ 6724.709463] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6724.714167] pending: vmstat_update [ 6724.717617] workqueue writeback: flags=0x4a [ 6724.720931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6724.728118] pending: wb_workfn, wb_workfn [ 6724.730920] workqueue xprtiod: flags=0xa [ 6724.732596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6724.735815] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6724.739410] workqueue ipv6_addrconf: flags=0x40008 [ 6724.741832] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6724.745664] pending: addrconf_verify_work [ 6724.747885] workqueue knbd0-recv: flags=0x1a [ 6724.750323] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6724.754296] in-flight: 647:recv_work [ 6724.756073] workqueue ptlrpc_pinger: flags=0x2 [ 6724.758032] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6724.761165] pending: ptlrpc_pinger_main [ptlrpc] [ 6724.764020] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6714s workers=2 idle: 652 [ 6736.059056] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6736.067174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6736.088842] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6736.092398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6736.095838] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6736.097887] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6736.104922] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6736.107809] RAX: 000000000000751e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6736.110392] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6736.113688] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6736.116079] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6736.119083] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6736.121692] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6736.124362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6736.128822] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6736.130740] Call Trace: [ 6736.131528] [ 6736.132352] ? show_regs.cold.9+0x22/0x2f [ 6736.133928] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6736.136156] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6736.138035] ? watchdog+0x40/0x40 [ 6736.139455] ? hrtimer_interrupt+0x128/0x2c0 [ 6736.141182] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6736.142741] ? apic_timer_interrupt+0xf/0x20 [ 6736.144084] [ 6736.144697] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6736.146249] do_raw_spin_lock+0x177/0x190 [ 6736.147910] _raw_spin_lock+0x21/0x30 [ 6736.149127] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6736.151161] ? do_raw_spin_unlock+0x75/0x190 [ 6736.153037] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6736.154856] ? do_wait_intr+0xf0/0xf0 [ 6736.156166] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6736.158110] mdc_close+0x53b/0xd70 [mdc] [ 6736.159836] lmv_close+0x18d/0x490 [lmv] [ 6736.161155] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6736.163813] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6736.166094] ll_md_close+0x274/0x800 [lustre] [ 6736.168392] ll_file_release+0x128/0xc50 [lustre] [ 6736.170461] __fput+0x10d/0x440 [ 6736.171842] ____fput+0x12/0x20 [ 6736.173244] task_work_run+0xc6/0x110 [ 6736.174857] exit_to_usermode_loop+0x1dd/0x1f0 [ 6736.176807] do_syscall_64+0x3d6/0x440 [ 6736.178557] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6736.180681] RIP: 0033:0x7f1a63983cd8 [ 6736.182153] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6736.192334] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6736.196371] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6736.200002] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6736.203560] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6736.207064] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6736.210658] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6748.056315] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6748.061028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6748.064989] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 6748.069255] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 6748.079563] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [ 6748.087716] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6748.087762] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6748.087769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6748.087771] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6748.087786] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6748.087790] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6748.087795] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6748.087797] RDX: 0000000000001cd5 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6748.087800] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6748.087802] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6748.087805] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6748.087809] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6748.087812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6748.087815] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6748.087824] Call Trace: [ 6748.087829] [ 6748.087835] ? show_regs.cold.9+0x22/0x2f [ 6748.087845] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6748.091745] ptlrpc_gss(O) [ 6748.103144] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6748.103163] ? watchdog+0x40/0x40 [ 6748.103169] ? hrtimer_interrupt+0x128/0x2c0 [ 6748.103173] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6748.103182] ? apic_timer_interrupt+0xf/0x20 [ 6748.103187] [ 6748.103192] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6748.103199] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6748.103204] do_raw_spin_lock+0x177/0x190 [ 6748.103211] _raw_spin_lock+0x21/0x30 [ 6748.103217] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6748.103794] obd_mod_free+0x46/0x140 [ptlrpc] [ 6748.104063] mdc_commit_open+0x91/0xb0 [mdc] [ 6748.104104] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6748.104302] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6748.104585] after_reply+0xe2c/0x1780 [ptlrpc] [ 6748.104880] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6748.110858] ptlrpc(O) [ 6748.117292] ? __next_timer_interrupt+0x160/0x160 [ 6748.117317] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6748.117690] ? do_wait_intr+0xf0/0xf0 [ 6748.117700] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6748.117899] kthread+0x1d1/0x200 [ 6748.117905] ? set_kthread_struct+0x70/0x70 [ 6748.117910] ret_from_fork+0x1f/0x30 [ 6748.219091] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6748.235976] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6748.245706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6748.249320] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6748.257321] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6748.268616] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6748.271550] RAX: 0000000000003093 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6748.274083] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6748.276572] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6748.278896] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6748.281673] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6748.284418] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6748.287434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6748.289888] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6748.292720] Call Trace: [ 6748.293863] [ 6748.294713] ? show_regs.cold.9+0x22/0x2f [ 6748.296348] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6748.298275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6748.300123] ? watchdog+0x40/0x40 [ 6748.301841] ? hrtimer_interrupt+0x128/0x2c0 [ 6748.303151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6748.304694] ? apic_timer_interrupt+0xf/0x20 [ 6748.306015] [ 6748.306909] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6748.308729] do_raw_spin_lock+0x177/0x190 [ 6748.310023] _raw_spin_lock+0x21/0x30 [ 6748.311116] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6748.313092] ? do_raw_spin_unlock+0x75/0x190 [ 6748.317597] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6748.320417] ? do_wait_intr+0xf0/0xf0 [ 6748.322145] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6748.325074] mdc_close+0x53b/0xd70 [mdc] [ 6748.329977] lmv_close+0x18d/0x490 [lmv] [ 6748.333430] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6748.337350] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6748.339234] ll_md_close+0x274/0x800 [lustre] [ 6748.342605] ll_file_release+0x128/0xc50 [lustre] [ 6748.345912] __fput+0x10d/0x440 [ 6748.347881] ____fput+0x12/0x20 [ 6748.349877] task_work_run+0xc6/0x110 [ 6748.352533] exit_to_usermode_loop+0x1dd/0x1f0 [ 6748.355371] do_syscall_64+0x3d6/0x440 [ 6748.358182] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6748.361181] RIP: 0033:0x7f1a63983cd8 [ 6748.363247] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6748.374652] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6748.380382] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6748.384773] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6748.389854] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6748.395403] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6748.400439] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6752.053029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 6752.056102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6752.089163] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6752.093605] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6752.103820] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6752.107177] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6752.118183] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6752.123285] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6752.129003] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6752.148182] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6752.154611] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6752.159148] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6752.163781] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6752.167609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6752.170402] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6752.173574] Call Trace: [ 6752.175009] [ 6752.177408] ? show_regs.cold.9+0x22/0x2f [ 6752.181895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6752.190981] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6752.205788] ? watchdog+0x40/0x40 [ 6752.207662] ? hrtimer_interrupt+0x128/0x2c0 [ 6752.213366] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6752.215157] ? apic_timer_interrupt+0xf/0x20 [ 6752.221320] [ 6752.223365] ? native_safe_halt+0x2e/0x50 [ 6752.225272] kvm_wait+0xcd/0xe0 [ 6752.232093] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6752.235302] do_raw_spin_lock+0x177/0x190 [ 6752.238224] _raw_spin_lock+0x21/0x30 [ 6752.240510] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6752.243517] ? do_raw_spin_unlock+0x75/0x190 [ 6752.246135] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6752.252587] ? do_wait_intr+0xf0/0xf0 [ 6752.254006] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6752.272843] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6752.283270] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6752.287102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6752.289114] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6752.297084] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6752.299168] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6752.301366] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6752.304080] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6752.311638] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6752.314750] lmv_intent_open+0x338/0x1150 [lmv] [ 6752.320229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6752.327254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6752.330959] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6752.339316] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6752.343297] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6752.352907] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6752.354502] ? do_raw_spin_unlock+0x75/0x190 [ 6752.364625] ? _raw_spin_unlock+0x12/0x30 [ 6752.371081] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6752.378289] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6752.382252] ll_atomic_open+0x36a/0xd70 [lustre] [ 6752.386186] lookup_open+0x356/0xb40 [ 6752.388828] do_last+0x313/0xfc0 [ 6752.391362] ? nd_jump_root+0xe5/0x160 [ 6752.398282] ? path_init+0x437/0x520 [ 6752.400224] path_openat+0xf7/0x500 [ 6752.406291] ? do_raw_spin_unlock+0x75/0x190 [ 6752.409859] do_filp_open+0x99/0x140 [ 6752.412223] ? getname_flags+0x6e/0x330 [ 6752.418705] ? __check_object_size+0xff/0x256 [ 6752.424219] ? do_raw_spin_unlock+0x75/0x190 [ 6752.426815] ? _raw_spin_unlock+0x12/0x30 [ 6752.431701] do_sys_openat2+0x2b4/0x410 [ 6752.434041] do_sys_open+0x73/0xa0 [ 6752.436633] __x64_sys_openat+0x24/0x30 [ 6752.439033] do_syscall_64+0xc1/0x440 [ 6752.441740] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6752.447342] RIP: 0033:0x7f1a63983332 [ 6752.452323] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6752.471754] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6752.490056] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6752.500533] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6752.509507] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6752.519809] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6752.527262] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6754.784075] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4731s! [ 6754.793846] Showing busy workqueues and worker pools: [ 6754.800719] workqueue events: flags=0x0 [ 6754.803095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6754.811754] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6754.820317] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6754.826044] pending: sec_gc_main [ptlrpc] [ 6754.828023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6754.830578] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6754.834852] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6754.837232] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6754.840732] workqueue events_long: flags=0x0 [ 6754.842592] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6754.845215] pending: lu_objects_flush [obdclass] [ 6754.847319] workqueue events_unbound: flags=0x2 [ 6754.849364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6754.851933] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6754.856364] workqueue events_power_efficient: flags=0x80 [ 6754.858629] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6754.861346] pending: neigh_periodic_work, do_cache_clean [ 6754.863607] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6754.866310] pending: neigh_periodic_work, check_lifetime [ 6754.868720] workqueue mm_percpu_wq: flags=0x8 [ 6754.871107] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6754.875226] pending: vmstat_update [ 6754.877041] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6754.879178] pending: vmstat_update [ 6754.880048] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6754.883789] pending: vmstat_update [ 6754.886157] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6754.887971] pending: vmstat_update [ 6754.888969] workqueue writeback: flags=0x4a [ 6754.890333] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6754.894098] pending: wb_workfn, wb_workfn [ 6754.895386] workqueue xprtiod: flags=0xa [ 6754.896773] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6754.899786] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6754.901886] workqueue ipv6_addrconf: flags=0x40008 [ 6754.904285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6754.907206] pending: addrconf_verify_work [ 6754.909759] workqueue knbd0-recv: flags=0x1a [ 6754.912089] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6754.914834] in-flight: 647:recv_work [ 6754.915880] workqueue ptlrpc_pinger: flags=0x2 [ 6754.917025] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6754.919851] pending: ptlrpc_pinger_main [ptlrpc] [ 6754.921709] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6744s workers=2 idle: 652 [ 6762.518146] rcu: INFO: rcu_sched self-detected stall on CPU [ 6762.523708] rcu: 3-...!: (2938925 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 6762.532785] (t=4740126 jiffies g=269633 q=1802) [ 6762.535801] rcu: rcu_sched kthread starved for 4739775 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6762.543711] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6762.548759] rcu: RCU grace-period kthread stack dump: [ 6762.551145] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6762.556578] Call Trace: [ 6762.557656] __schedule+0x351/0xcb0 [ 6762.559395] schedule+0xc0/0x180 [ 6762.560869] schedule_timeout+0xb4/0x190 [ 6762.562825] ? do_raw_spin_unlock+0x75/0x190 [ 6762.566229] ? __next_timer_interrupt+0x160/0x160 [ 6762.568706] ? prepare_to_swait_event+0xf3/0x140 [ 6762.574613] rcu_gp_kthread+0x8dd/0x15b0 [ 6762.578363] ? force_qs_rnp+0x2c0/0x2c0 [ 6762.581867] kthread+0x1d1/0x200 [ 6762.583629] ? set_kthread_struct+0x70/0x70 [ 6762.588429] ret_from_fork+0x1f/0x30 [ 6762.589741] rcu: Stack dump where RCU GP kthread last ran: [ 6762.600766] Sending NMI from CPU 3 to CPUs 1: [ 6762.605083] NMI backtrace for cpu 1 [ 6762.605089] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6762.605091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6762.605093] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6762.605094] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6762.605096] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 [ 6762.605099] RAX: 00000000000040c2 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6762.605100] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6762.605101] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6762.605102] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6762.605104] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6762.605105] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6762.605106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6762.605107] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6762.605109] Call Trace: [ 6762.605110] [ 6762.605111] ? show_regs.cold.9+0x22/0x2f [ 6762.605112] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6762.605113] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6762.605114] ? nmi_handle+0xbd/0x210 [ 6762.605114] ? default_do_nmi+0x4b/0x1b0 [ 6762.605115] ? do_nmi+0x280/0x380 [ 6762.605116] ? end_repeat_nmi+0x16/0x1f [ 6762.605117] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6762.605118] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6762.605120] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6762.605121] [ 6762.605121] do_raw_spin_lock+0x177/0x190 [ 6762.605123] _raw_spin_lock+0x21/0x30 [ 6762.605123] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6762.605124] ? do_raw_spin_unlock+0x75/0x190 [ 6762.605125] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6762.605126] ? do_wait_intr+0xf0/0xf0 [ 6762.605127] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6762.605128] mdc_close+0x53b/0xd70 [mdc] [ 6762.605129] lmv_close+0x18d/0x490 [lmv] [ 6762.605130] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6762.605131] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6762.605133] ll_md_close+0x274/0x800 [lustre] [ 6762.605133] ll_file_release+0x128/0xc50 [lustre] [ 6762.605134] __fput+0x10d/0x440 [ 6762.605135] ____fput+0x12/0x20 [ 6762.605136] task_work_run+0xc6/0x110 [ 6762.605137] exit_to_usermode_loop+0x1dd/0x1f0 [ 6762.605138] do_syscall_64+0x3d6/0x440 [ 6762.605139] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6762.605140] RIP: 0033:0x7f1a63983cd8 [ 6762.605141] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6762.605144] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6762.605147] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6762.605148] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6762.605150] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6762.605151] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6762.605152] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6762.608068] NMI backtrace for cpu 3 [ 6762.864689] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6762.871764] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6762.877953] Call Trace: [ 6762.880047] [ 6762.881963] ? dump_stack+0xbb/0x10e [ 6762.885763] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6762.889977] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6762.893936] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6762.898069] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6762.902689] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6762.908562] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6762.912840] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6762.918112] ? update_process_times+0x6d/0xc0 [ 6762.930652] ? tick_sched_handle.isra.18+0x27/0x90 [ 6762.932200] ? tick_sched_timer+0x53/0xb0 [ 6762.945574] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6762.946961] ? tick_sched_handle.isra.18+0x90/0x90 [ 6762.961130] ? hrtimer_interrupt+0x128/0x2c0 [ 6762.971880] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6762.977742] ? apic_timer_interrupt+0xf/0x20 [ 6762.979248] [ 6762.986686] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6762.989670] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6762.992084] ? do_raw_spin_lock+0x177/0x190 [ 6762.996254] ? _raw_spin_lock+0x21/0x30 [ 6762.997680] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6763.006189] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6763.009313] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6763.013715] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6763.016862] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6763.020957] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6763.024064] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6763.028446] ? __next_timer_interrupt+0x160/0x160 [ 6763.032186] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6763.036088] ? do_wait_intr+0xf0/0xf0 [ 6763.040074] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6763.042226] ? kthread+0x1d1/0x200 [ 6763.047746] ? set_kthread_struct+0x70/0x70 [ 6763.049203] ? ret_from_fork+0x1f/0x30 [ 6764.060492] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6764.065020] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6764.089662] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6764.094911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6764.099432] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6764.102368] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6764.114127] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6764.120994] RAX: 000000000000484a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6764.129900] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6764.134703] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6764.139026] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6764.143109] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6764.145991] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6764.150200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6764.153305] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6764.157700] Call Trace: [ 6764.159180] [ 6764.160191] ? show_regs.cold.9+0x22/0x2f [ 6764.162456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6764.165666] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6764.167915] ? watchdog+0x40/0x40 [ 6764.176181] ? hrtimer_interrupt+0x128/0x2c0 [ 6764.181044] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6764.185883] ? apic_timer_interrupt+0xf/0x20 [ 6764.189067] [ 6764.190887] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6764.194146] do_raw_spin_lock+0x177/0x190 [ 6764.199916] _raw_spin_lock+0x21/0x30 [ 6764.202105] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6764.207664] ? do_raw_spin_unlock+0x75/0x190 [ 6764.210704] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6764.223483] ? do_wait_intr+0xf0/0xf0 [ 6764.229694] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6764.235125] mdc_close+0x53b/0xd70 [mdc] [ 6764.236725] lmv_close+0x18d/0x490 [lmv] [ 6764.239945] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6764.242939] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6764.245708] ll_md_close+0x274/0x800 [lustre] [ 6764.247867] ll_file_release+0x128/0xc50 [lustre] [ 6764.250374] __fput+0x10d/0x440 [ 6764.251463] ____fput+0x12/0x20 [ 6764.257241] task_work_run+0xc6/0x110 [ 6764.264206] exit_to_usermode_loop+0x1dd/0x1f0 [ 6764.269682] do_syscall_64+0x3d6/0x440 [ 6764.272076] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6764.274779] RIP: 0033:0x7f1a63983cd8 [ 6764.276275] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6764.285608] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6764.288113] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6764.299592] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6764.303870] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6764.306775] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6764.314591] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6780.053008] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6780.058729] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6780.089438] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6780.096611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6780.103853] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6780.107734] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 6780.118560] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6780.123347] RAX: 0000000000006963 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6780.127810] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6780.131896] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6780.135940] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6780.141319] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6780.145664] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6780.150360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6780.153421] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6780.157593] Call Trace: [ 6780.158702] [ 6780.159655] ? show_regs.cold.9+0x22/0x2f [ 6780.161299] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6780.163632] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6780.166121] ? watchdog+0x40/0x40 [ 6780.167998] ? hrtimer_interrupt+0x128/0x2c0 [ 6780.170335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6780.174276] ? apic_timer_interrupt+0xf/0x20 [ 6780.179932] [ 6780.181099] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6780.184574] do_raw_spin_lock+0x177/0x190 [ 6780.189977] _raw_spin_lock+0x21/0x30 [ 6780.193854] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6780.200909] ? do_raw_spin_unlock+0x75/0x190 [ 6780.203249] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6780.206316] ? do_wait_intr+0xf0/0xf0 [ 6780.207914] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6780.214730] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6780.221802] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6780.229042] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6780.230931] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6780.234227] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6780.237323] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6780.239600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6780.241681] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6780.252738] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6780.257976] lmv_intent_open+0x338/0x1150 [lmv] [ 6780.262927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6780.271379] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6780.276373] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6780.278901] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6780.286895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6780.295349] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6780.299676] ? do_raw_spin_unlock+0x75/0x190 [ 6780.302768] ? _raw_spin_unlock+0x12/0x30 [ 6780.304806] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6780.309709] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6780.316568] ll_atomic_open+0x36a/0xd70 [lustre] [ 6780.323513] lookup_open+0x356/0xb40 [ 6780.326137] do_last+0x313/0xfc0 [ 6780.329867] ? nd_jump_root+0xe5/0x160 [ 6780.334784] ? path_init+0x437/0x520 [ 6780.338303] path_openat+0xf7/0x500 [ 6780.340199] ? do_raw_spin_unlock+0x75/0x190 [ 6780.343888] do_filp_open+0x99/0x140 [ 6780.346658] ? getname_flags+0x6e/0x330 [ 6780.349265] ? __check_object_size+0xff/0x256 [ 6780.361798] ? do_raw_spin_unlock+0x75/0x190 [ 6780.367659] ? _raw_spin_unlock+0x12/0x30 [ 6780.372015] do_sys_openat2+0x2b4/0x410 [ 6780.380518] do_sys_open+0x73/0xa0 [ 6780.382306] __x64_sys_openat+0x24/0x30 [ 6780.388656] do_syscall_64+0xc1/0x440 [ 6780.390782] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6780.392083] RIP: 0033:0x7f1a63983332 [ 6780.397304] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6780.414831] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6780.424834] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6780.431150] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6780.436117] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6780.442620] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6780.452416] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6784.993446] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4761s! [ 6785.010738] Showing busy workqueues and worker pools: [ 6785.018373] workqueue events: flags=0x0 [ 6785.022686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6785.029852] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6785.044988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6785.053589] pending: sec_gc_main [ptlrpc] [ 6785.059685] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6785.070155] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6785.087921] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6785.094332] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6785.107905] workqueue events_long: flags=0x0 [ 6785.115941] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6785.126012] pending: lu_objects_flush [obdclass] [ 6785.132249] workqueue events_unbound: flags=0x2 [ 6785.140845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6785.147801] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6785.157088] workqueue events_power_efficient: flags=0x80 [ 6785.160837] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6785.167350] pending: neigh_periodic_work, do_cache_clean [ 6785.171226] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6785.174280] pending: neigh_periodic_work, check_lifetime [ 6785.176738] workqueue mm_percpu_wq: flags=0x8 [ 6785.178792] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6785.181750] pending: vmstat_update [ 6785.183351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6785.186697] pending: vmstat_update [ 6785.188621] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6785.193083] pending: vmstat_update [ 6785.196277] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6785.201853] pending: vmstat_update [ 6785.204941] workqueue writeback: flags=0x4a [ 6785.206383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6785.208909] pending: wb_workfn, wb_workfn [ 6785.210708] workqueue xprtiod: flags=0xa [ 6785.212342] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6785.215463] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6785.221585] workqueue ipv6_addrconf: flags=0x40008 [ 6785.225753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6785.229389] pending: addrconf_verify_work [ 6785.233365] workqueue knbd0-recv: flags=0x1a [ 6785.237954] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6785.242067] in-flight: 647:recv_work [ 6785.246558] workqueue ptlrpc_pinger: flags=0x2 [ 6785.251688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6785.255817] pending: ptlrpc_pinger_main [ptlrpc] [ 6785.259375] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6775s workers=2 idle: 652 [ 6788.059212] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6788.059464] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6788.063939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6788.064019] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6788.064026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6788.064028] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6788.064043] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6788.064047] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6788.064052] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6788.064054] RDX: 0000000000000fbf RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6788.064057] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6788.064059] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6788.064062] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6788.064065] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6788.064068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6788.064070] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6788.064080] Call Trace: [ 6788.064094] [ 6788.064100] ? show_regs.cold.9+0x22/0x2f [ 6788.064110] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6788.064117] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6788.064122] ? watchdog+0x40/0x40 [ 6788.064126] ? hrtimer_interrupt+0x128/0x2c0 [ 6788.064130] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6788.064138] ? apic_timer_interrupt+0xf/0x20 [ 6788.064143] [ 6788.064146] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6788.064152] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6788.064156] do_raw_spin_lock+0x177/0x190 [ 6788.064161] _raw_spin_lock+0x21/0x30 [ 6788.064167] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6788.070664] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6788.070871] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6788.070879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6788.070881] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6788.070900] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6788.070904] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6788.070910] RAX: 0000000000007f1d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6788.070912] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6788.070914] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6788.070916] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6788.070919] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6788.070922] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6788.070926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6788.070928] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6788.070936] Call Trace: [ 6788.070942] [ 6788.070947] ? show_regs.cold.9+0x22/0x2f [ 6788.070957] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6788.070963] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6788.070967] ? watchdog+0x40/0x40 [ 6788.070972] ? hrtimer_interrupt+0x128/0x2c0 [ 6788.070977] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6788.070985] ? apic_timer_interrupt+0xf/0x20 [ 6788.070990] [ 6788.070992] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6788.070999] do_raw_spin_lock+0x177/0x190 [ 6788.071004] _raw_spin_lock+0x21/0x30 [ 6788.071010] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6788.071455] ? do_raw_spin_unlock+0x75/0x190 [ 6788.071462] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6788.071687] ? do_wait_intr+0xf0/0xf0 [ 6788.071695] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6788.071908] mdc_close+0x53b/0xd70 [mdc] [ 6788.073043] lmv_close+0x18d/0x490 [lmv] [ 6788.073074] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6788.073342] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6788.073431] ll_md_close+0x274/0x800 [lustre] [ 6788.073520] ll_file_release+0x128/0xc50 [lustre] [ 6788.073614] __fput+0x10d/0x440 [ 6788.073624] ____fput+0x12/0x20 [ 6788.073628] task_work_run+0xc6/0x110 [ 6788.073641] exit_to_usermode_loop+0x1dd/0x1f0 [ 6788.073649] do_syscall_64+0x3d6/0x440 [ 6788.073654] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6788.073662] RIP: 0033:0x7f1a63983cd8 [ 6788.073672] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6788.073676] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6788.073681] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6788.073686] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6788.073688] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6788.073691] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6788.073693] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6788.431300] obd_mod_free+0x46/0x140 [ptlrpc] [ 6788.434901] mdc_commit_open+0x91/0xb0 [mdc] [ 6788.436826] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6788.439616] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6788.442392] after_reply+0xe2c/0x1780 [ptlrpc] [ 6788.444906] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6788.447594] ? __next_timer_interrupt+0x160/0x160 [ 6788.449400] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6788.452003] ? do_wait_intr+0xf0/0xf0 [ 6788.453970] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6788.456669] kthread+0x1d1/0x200 [ 6788.458258] ? set_kthread_struct+0x70/0x70 [ 6788.460625] ret_from_fork+0x1f/0x30 [ 6792.057036] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6792.060974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6792.081931] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6792.087886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6792.093183] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6792.101401] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6792.118841] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6792.122729] RAX: 000000000000530c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6792.132557] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6792.139561] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6792.149930] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6792.156988] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6792.165008] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6792.176698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6792.183922] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6792.189671] Call Trace: [ 6792.190577] [ 6792.193684] ? show_regs.cold.9+0x22/0x2f [ 6792.196263] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6792.200082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6792.202976] ? watchdog+0x40/0x40 [ 6792.206057] ? hrtimer_interrupt+0x128/0x2c0 [ 6792.207769] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6792.216528] ? apic_timer_interrupt+0xf/0x20 [ 6792.221641] [ 6792.223512] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6792.229261] do_raw_spin_lock+0x177/0x190 [ 6792.232325] _raw_spin_lock+0x21/0x30 [ 6792.239554] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6792.243930] ? do_raw_spin_unlock+0x75/0x190 [ 6792.247459] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6792.251672] ? do_wait_intr+0xf0/0xf0 [ 6792.269453] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6792.280558] mdc_close+0x53b/0xd70 [mdc] [ 6792.293328] lmv_close+0x18d/0x490 [lmv] [ 6792.298917] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6792.323537] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6792.333805] ll_md_close+0x274/0x800 [lustre] [ 6792.347455] ll_file_release+0x128/0xc50 [lustre] [ 6792.354515] __fput+0x10d/0x440 [ 6792.369461] ____fput+0x12/0x20 [ 6792.379784] task_work_run+0xc6/0x110 [ 6792.394657] exit_to_usermode_loop+0x1dd/0x1f0 [ 6792.401302] do_syscall_64+0x3d6/0x440 [ 6792.409735] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6792.426333] RIP: 0033:0x7f1a63983cd8 [ 6792.430946] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6792.468889] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6792.480502] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6792.494594] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6792.509793] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6792.521318] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6792.536960] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6812.053026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 6812.060881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6812.081468] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6812.093957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6812.098289] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6812.102991] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6812.120421] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6812.129456] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6812.136517] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6812.142108] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6812.149379] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6812.155772] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6812.160471] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6812.166465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6812.170235] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6812.175089] Call Trace: [ 6812.176654] [ 6812.183880] ? show_regs.cold.9+0x22/0x2f [ 6812.189056] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6812.195106] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6812.196785] ? watchdog+0x40/0x40 [ 6812.198997] ? hrtimer_interrupt+0x128/0x2c0 [ 6812.203077] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6812.205062] ? apic_timer_interrupt+0xf/0x20 [ 6812.208088] [ 6812.209220] ? native_safe_halt+0x2e/0x50 [ 6812.212416] kvm_wait+0xcd/0xe0 [ 6812.214315] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6812.217569] do_raw_spin_lock+0x177/0x190 [ 6812.220660] _raw_spin_lock+0x21/0x30 [ 6812.223054] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6812.227694] ? do_raw_spin_unlock+0x75/0x190 [ 6812.231422] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6812.236323] ? do_wait_intr+0xf0/0xf0 [ 6812.237763] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6812.241437] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6812.244523] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6812.249317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6812.252448] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6812.256572] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6812.258389] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6812.261801] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6812.266102] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6812.270532] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6812.273324] lmv_intent_open+0x338/0x1150 [lmv] [ 6812.277358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6812.281604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6812.295017] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6812.296637] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6812.299799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6812.305623] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6812.308861] ? do_raw_spin_unlock+0x75/0x190 [ 6812.312801] ? _raw_spin_unlock+0x12/0x30 [ 6812.316381] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6812.321950] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6812.327122] ll_atomic_open+0x36a/0xd70 [lustre] [ 6812.332686] lookup_open+0x356/0xb40 [ 6812.335486] do_last+0x313/0xfc0 [ 6812.337569] ? nd_jump_root+0xe5/0x160 [ 6812.347390] ? path_init+0x437/0x520 [ 6812.351454] path_openat+0xf7/0x500 [ 6812.352806] ? do_raw_spin_unlock+0x75/0x190 [ 6812.358637] do_filp_open+0x99/0x140 [ 6812.361739] ? getname_flags+0x6e/0x330 [ 6812.365226] ? __check_object_size+0xff/0x256 [ 6812.372791] ? do_raw_spin_unlock+0x75/0x190 [ 6812.375804] ? _raw_spin_unlock+0x12/0x30 [ 6812.379818] do_sys_openat2+0x2b4/0x410 [ 6812.387690] do_sys_open+0x73/0xa0 [ 6812.391527] __x64_sys_openat+0x24/0x30 [ 6812.394026] do_syscall_64+0xc1/0x440 [ 6812.396883] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6812.401488] RIP: 0033:0x7f1a63983332 [ 6812.405017] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6812.420065] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6812.425625] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6812.427783] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6812.431517] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6812.435637] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6812.442499] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6815.717222] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4792s! [ 6815.728694] Showing busy workqueues and worker pools: [ 6815.731037] workqueue events: flags=0x0 [ 6815.735396] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6815.743760] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6815.751326] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6815.758463] pending: sec_gc_main [ptlrpc] [ 6815.766732] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6815.774104] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6815.785974] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6815.795782] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6815.806184] workqueue events_long: flags=0x0 [ 6815.810260] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6815.816893] pending: lu_objects_flush [obdclass] [ 6815.823357] workqueue events_unbound: flags=0x2 [ 6815.827482] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6815.834345] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6815.845351] workqueue events_power_efficient: flags=0x80 [ 6815.850878] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6815.854272] pending: neigh_periodic_work, do_cache_clean [ 6815.858938] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6815.862652] pending: neigh_periodic_work, check_lifetime [ 6815.864877] workqueue mm_percpu_wq: flags=0x8 [ 6815.869096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6815.876776] pending: vmstat_update [ 6815.881399] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6815.885358] pending: vmstat_update [ 6815.890339] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6815.894681] pending: vmstat_update [ 6815.896959] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6815.900665] pending: vmstat_update [ 6815.902213] workqueue writeback: flags=0x4a [ 6815.911071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6815.914983] pending: wb_workfn, wb_workfn [ 6815.919071] workqueue xprtiod: flags=0xa [ 6815.922237] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6815.928743] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6815.937793] workqueue ipv6_addrconf: flags=0x40008 [ 6815.943135] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6815.949982] pending: addrconf_verify_work [ 6815.953486] workqueue knbd0-recv: flags=0x1a [ 6815.957187] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6815.963088] in-flight: 647:recv_work [ 6815.965100] workqueue ptlrpc_pinger: flags=0x2 [ 6815.966839] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6815.969732] pending: ptlrpc_pinger_main [ptlrpc] [ 6815.972427] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6806s workers=2 idle: 652 [ 6816.056598] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6816.058059] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6816.059071] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6816.059145] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6816.059151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6816.059154] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6816.059167] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6816.059171] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6816.059175] RAX: 0000000000000683 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6816.059178] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6816.059180] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6816.059182] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6816.059185] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6816.059188] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6816.059191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6816.059194] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6816.059202] Call Trace: [ 6816.059207] [ 6816.059212] ? show_regs.cold.9+0x22/0x2f [ 6816.059222] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6816.059230] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6816.059235] ? watchdog+0x40/0x40 [ 6816.059239] ? hrtimer_interrupt+0x128/0x2c0 [ 6816.059243] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6816.059251] ? apic_timer_interrupt+0xf/0x20 [ 6816.059255] [ 6816.059259] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6816.059264] do_raw_spin_lock+0x177/0x190 [ 6816.059269] _raw_spin_lock+0x21/0x30 [ 6816.059273] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6816.063214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [ 6816.093471] ? do_raw_spin_unlock+0x75/0x190 [ 6816.093492] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6816.093974] ? do_wait_intr+0xf0/0xf0 [ 6816.093983] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6816.094175] mdc_close+0x53b/0xd70 [mdc] [ 6816.101686] sha512_ssse3 sha512_generic [ 6816.101702] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6816.101710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6816.101713] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6816.101726] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6816.101730] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6816.101734] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6816.101747] RDX: 0000000000006794 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6816.101750] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6816.101753] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6816.101755] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6816.101759] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6816.101762] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6816.101765] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6816.101772] Call Trace: [ 6816.101777] [ 6816.101782] ? show_regs.cold.9+0x22/0x2f [ 6816.101791] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6816.101797] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6816.101802] ? watchdog+0x40/0x40 [ 6816.101807] ? hrtimer_interrupt+0x128/0x2c0 [ 6816.101811] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6816.101830] ? apic_timer_interrupt+0xf/0x20 [ 6816.101834] [ 6816.101837] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6816.101842] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6816.101846] do_raw_spin_lock+0x177/0x190 [ 6816.101851] _raw_spin_lock+0x21/0x30 [ 6816.101912] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6816.102209] obd_mod_free+0x46/0x140 [ptlrpc] [ 6816.102426] mdc_commit_open+0x91/0xb0 [mdc] [ 6816.102464] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6816.102833] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6816.108465] lmv_close+0x18d/0x490 [lmv] [ 6816.112076] after_reply+0xe2c/0x1780 [ptlrpc] [ 6816.128139] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6816.128273] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6816.128354] ll_md_close+0x274/0x800 [lustre] [ 6816.128436] ll_file_release+0x128/0xc50 [lustre] [ 6816.135709] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6816.136451] ? __next_timer_interrupt+0x160/0x160 [ 6816.136461] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6816.148910] __fput+0x10d/0x440 [ 6816.148934] ____fput+0x12/0x20 [ 6816.148939] task_work_run+0xc6/0x110 [ 6816.148947] exit_to_usermode_loop+0x1dd/0x1f0 [ 6816.148955] do_syscall_64+0x3d6/0x440 [ 6816.148960] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6816.148970] RIP: 0033:0x7f1a63983cd8 [ 6816.148979] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6816.148984] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6816.148990] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6816.148992] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6816.148994] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6816.148998] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6816.149001] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6816.556775] ? do_wait_intr+0xf0/0xf0 [ 6816.558778] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6816.561568] kthread+0x1d1/0x200 [ 6816.562402] ? set_kthread_struct+0x70/0x70 [ 6816.563661] ret_from_fork+0x1f/0x30 [ 6820.060336] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6820.068476] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6820.098520] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6820.103759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6820.109949] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6820.113748] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6820.128404] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6820.131079] RAX: 000000000000291a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6820.133587] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6820.136353] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6820.138362] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6820.140688] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6820.142305] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6820.145050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6820.147122] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6820.149175] Call Trace: [ 6820.150150] [ 6820.150718] ? show_regs.cold.9+0x22/0x2f [ 6820.152045] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6820.153432] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6820.154756] ? watchdog+0x40/0x40 [ 6820.156148] ? hrtimer_interrupt+0x128/0x2c0 [ 6820.157619] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6820.159264] ? apic_timer_interrupt+0xf/0x20 [ 6820.161447] [ 6820.162516] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6820.167379] do_raw_spin_lock+0x177/0x190 [ 6820.168968] _raw_spin_lock+0x21/0x30 [ 6820.171089] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6820.174858] ? do_raw_spin_unlock+0x75/0x190 [ 6820.177295] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6820.181120] ? do_wait_intr+0xf0/0xf0 [ 6820.183117] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6820.186186] mdc_close+0x53b/0xd70 [mdc] [ 6820.187704] lmv_close+0x18d/0x490 [lmv] [ 6820.190631] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6820.193368] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6820.199852] ll_md_close+0x274/0x800 [lustre] [ 6820.204211] ll_file_release+0x128/0xc50 [lustre] [ 6820.206948] __fput+0x10d/0x440 [ 6820.208848] ____fput+0x12/0x20 [ 6820.210551] task_work_run+0xc6/0x110 [ 6820.219404] exit_to_usermode_loop+0x1dd/0x1f0 [ 6820.226049] do_syscall_64+0x3d6/0x440 [ 6820.227940] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6820.229441] RIP: 0033:0x7f1a63983cd8 [ 6820.230704] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6820.238063] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6820.242376] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6820.246448] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6820.249929] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6820.254169] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6820.259917] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6840.053032] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 6840.073051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6840.134464] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6840.151854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6840.160002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6840.170406] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6840.198483] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6840.204241] RAX: 0000000000006187 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6840.209229] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6840.213325] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6840.217284] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6840.220889] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6840.223902] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6840.226949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6840.229219] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6840.231848] Call Trace: [ 6840.232871] [ 6840.233718] ? show_regs.cold.9+0x22/0x2f [ 6840.234969] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6840.236960] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6840.238694] ? watchdog+0x40/0x40 [ 6840.239920] ? hrtimer_interrupt+0x128/0x2c0 [ 6840.241302] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6840.243030] ? apic_timer_interrupt+0xf/0x20 [ 6840.244309] [ 6840.244937] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6840.251024] do_raw_spin_lock+0x177/0x190 [ 6840.256386] _raw_spin_lock+0x21/0x30 [ 6840.261520] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6840.273285] ? do_raw_spin_unlock+0x75/0x190 [ 6840.275040] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6840.278219] ? do_wait_intr+0xf0/0xf0 [ 6840.286430] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6840.291751] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6840.300308] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6840.303041] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6840.305401] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6840.308035] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6840.312953] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6840.315330] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6840.318073] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6840.320691] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6840.323304] lmv_intent_open+0x338/0x1150 [lmv] [ 6840.325368] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6840.327742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6840.330235] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6840.332431] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6840.334707] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6840.336948] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6840.339004] ? do_raw_spin_unlock+0x75/0x190 [ 6840.341485] ? _raw_spin_unlock+0x12/0x30 [ 6840.344053] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6840.348064] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6840.351185] ll_atomic_open+0x36a/0xd70 [lustre] [ 6840.374914] lookup_open+0x356/0xb40 [ 6840.377954] do_last+0x313/0xfc0 [ 6840.379317] ? nd_jump_root+0xe5/0x160 [ 6840.390371] ? path_init+0x437/0x520 [ 6840.392004] path_openat+0xf7/0x500 [ 6840.393140] ? do_raw_spin_unlock+0x75/0x190 [ 6840.394836] do_filp_open+0x99/0x140 [ 6840.396276] ? getname_flags+0x6e/0x330 [ 6840.398963] ? __check_object_size+0xff/0x256 [ 6840.401223] ? do_raw_spin_unlock+0x75/0x190 [ 6840.404196] ? _raw_spin_unlock+0x12/0x30 [ 6840.406044] do_sys_openat2+0x2b4/0x410 [ 6840.408129] do_sys_open+0x73/0xa0 [ 6840.423594] __x64_sys_openat+0x24/0x30 [ 6840.425341] do_syscall_64+0xc1/0x440 [ 6840.426984] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6840.429766] RIP: 0033:0x7f1a63983332 [ 6840.431420] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6840.439702] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6840.442667] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6840.450638] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6840.464331] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6840.481215] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6840.489379] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6844.056929] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 6844.058043] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6844.064577] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6844.064685] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6844.064692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6844.064695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6844.064713] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6844.064718] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6844.064723] RAX: 0000000000006388 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6844.064726] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6844.064729] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6844.064731] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6844.064734] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6844.064737] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6844.064740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6844.064743] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6844.064753] Call Trace: [ 6844.064758] [ 6844.064764] ? show_regs.cold.9+0x22/0x2f [ 6844.064774] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6844.064780] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6844.064785] ? watchdog+0x40/0x40 [ 6844.064790] ? hrtimer_interrupt+0x128/0x2c0 [ 6844.064794] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6844.064804] ? apic_timer_interrupt+0xf/0x20 [ 6844.064809] [ 6844.064812] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6844.064886] do_raw_spin_lock+0x177/0x190 [ 6844.064894] _raw_spin_lock+0x21/0x30 [ 6844.064900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6844.071910] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6844.072437] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6844.072444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6844.072448] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6844.072464] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6844.072468] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6844.072474] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6844.072476] RDX: 0000000000006c51 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6844.072478] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6844.072480] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6844.072482] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6844.072486] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6844.072489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6844.072492] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6844.072501] Call Trace: [ 6844.072506] [ 6844.072514] ? show_regs.cold.9+0x22/0x2f [ 6844.072525] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6844.072532] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6844.072536] ? watchdog+0x40/0x40 [ 6844.072555] ? hrtimer_interrupt+0x128/0x2c0 [ 6844.072560] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6844.072568] ? apic_timer_interrupt+0xf/0x20 [ 6844.072573] [ 6844.072576] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6844.072581] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6844.072585] do_raw_spin_lock+0x177/0x190 [ 6844.072590] _raw_spin_lock+0x21/0x30 [ 6844.072595] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6844.072914] obd_mod_free+0x46/0x140 [ptlrpc] [ 6844.073367] mdc_commit_open+0x91/0xb0 [mdc] [ 6844.073406] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6844.073620] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6844.073815] after_reply+0xe2c/0x1780 [ptlrpc] [ 6844.117739] ? do_raw_spin_unlock+0x75/0x190 [ 6844.132337] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6844.132623] ? __next_timer_interrupt+0x160/0x160 [ 6844.132634] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6844.132986] ? do_wait_intr+0xf0/0xf0 [ 6844.132996] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6844.133209] kthread+0x1d1/0x200 [ 6844.133215] ? set_kthread_struct+0x70/0x70 [ 6844.133219] ret_from_fork+0x1f/0x30 [ 6844.607103] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6844.610130] ? do_wait_intr+0xf0/0xf0 [ 6844.611811] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6844.618697] mdc_close+0x53b/0xd70 [mdc] [ 6844.622945] lmv_close+0x18d/0x490 [lmv] [ 6844.628513] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6844.631400] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6844.633787] ll_md_close+0x274/0x800 [lustre] [ 6844.636125] ll_file_release+0x128/0xc50 [lustre] [ 6844.639886] __fput+0x10d/0x440 [ 6844.644064] ____fput+0x12/0x20 [ 6844.646136] task_work_run+0xc6/0x110 [ 6844.649394] exit_to_usermode_loop+0x1dd/0x1f0 [ 6844.653225] do_syscall_64+0x3d6/0x440 [ 6844.655293] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6844.658743] RIP: 0033:0x7f1a63983cd8 [ 6844.660826] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6844.673190] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6844.678284] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6844.681819] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6844.686592] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6844.693502] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6844.697663] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6846.434742] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4823s! [ 6846.438247] Showing busy workqueues and worker pools: [ 6846.447977] workqueue events: flags=0x0 [ 6846.451407] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6846.459001] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6846.462929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6846.468700] pending: sec_gc_main [ptlrpc] [ 6846.473862] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6846.479074] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6846.490451] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6846.494479] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6846.504467] workqueue events_long: flags=0x0 [ 6846.507089] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6846.509749] pending: lu_objects_flush [obdclass] [ 6846.511941] workqueue events_unbound: flags=0x2 [ 6846.513836] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6846.516835] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6846.522244] workqueue events_power_efficient: flags=0x80 [ 6846.525481] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6846.534994] pending: neigh_periodic_work, do_cache_clean [ 6846.540906] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6846.547827] pending: neigh_periodic_work, check_lifetime [ 6846.555030] workqueue mm_percpu_wq: flags=0x8 [ 6846.559819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6846.569208] pending: vmstat_update [ 6846.573553] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6846.577917] pending: vmstat_update [ 6846.581677] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6846.589870] pending: vmstat_update [ 6846.595366] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6846.597722] pending: vmstat_update [ 6846.614029] workqueue writeback: flags=0x4a [ 6846.621888] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6846.627077] pending: wb_workfn, wb_workfn [ 6846.636823] workqueue xprtiod: flags=0xa [ 6846.643398] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6846.652011] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6846.671168] workqueue ipv6_addrconf: flags=0x40008 [ 6846.672919] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6846.700865] pending: addrconf_verify_work [ 6846.707925] workqueue knbd0-recv: flags=0x1a [ 6846.711713] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6846.718886] in-flight: 647:recv_work [ 6846.730048] workqueue ptlrpc_pinger: flags=0x2 [ 6846.739084] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6846.749642] pending: ptlrpc_pinger_main [ptlrpc] [ 6846.765224] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6836s workers=2 idle: 652 [ 6848.057029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6848.061360] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6848.089294] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6848.093386] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6848.096193] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6848.098429] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6848.105750] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6848.108357] RAX: 0000000000003a26 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6848.111070] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6848.113673] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6848.115856] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6848.118416] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6848.121057] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6848.124058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6848.126177] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6848.128697] Call Trace: [ 6848.129591] [ 6848.130186] ? show_regs.cold.9+0x22/0x2f [ 6848.131611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6848.133141] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6848.134617] ? watchdog+0x40/0x40 [ 6848.135968] ? hrtimer_interrupt+0x128/0x2c0 [ 6848.137459] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6848.139013] ? apic_timer_interrupt+0xf/0x20 [ 6848.140457] [ 6848.141110] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6848.143271] do_raw_spin_lock+0x177/0x190 [ 6848.144608] _raw_spin_lock+0x21/0x30 [ 6848.145808] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6848.148047] ? do_raw_spin_unlock+0x75/0x190 [ 6848.151047] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6848.165131] ? do_wait_intr+0xf0/0xf0 [ 6848.167446] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6848.172443] mdc_close+0x53b/0xd70 [mdc] [ 6848.174993] lmv_close+0x18d/0x490 [lmv] [ 6848.180843] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6848.187473] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6848.190492] ll_md_close+0x274/0x800 [lustre] [ 6848.193344] ll_file_release+0x128/0xc50 [lustre] [ 6848.196469] __fput+0x10d/0x440 [ 6848.199168] ____fput+0x12/0x20 [ 6848.201119] task_work_run+0xc6/0x110 [ 6848.203038] exit_to_usermode_loop+0x1dd/0x1f0 [ 6848.205117] do_syscall_64+0x3d6/0x440 [ 6848.207309] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6848.209850] RIP: 0033:0x7f1a63983cd8 [ 6848.211433] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6848.220977] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6848.224096] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6848.227369] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6848.232604] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6848.236796] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6848.239156] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6872.053038] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 6872.056620] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 6872.058037] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6872.058055] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6872.060293] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6872.060300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6872.060303] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6872.060326] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6872.060331] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6872.060335] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6872.060338] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6872.060340] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6872.060342] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6872.060345] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6872.060348] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6872.060350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6872.060352] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6872.060360] Call Trace: [ 6872.060375] [ 6872.060381] ? show_regs.cold.9+0x22/0x2f [ 6872.060390] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6872.060398] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6872.060402] ? watchdog+0x40/0x40 [ 6872.060407] ? hrtimer_interrupt+0x128/0x2c0 [ 6872.060411] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6872.060418] ? apic_timer_interrupt+0xf/0x20 [ 6872.060423] [ 6872.060426] ? native_safe_halt+0x2e/0x50 [ 6872.060431] kvm_wait+0xcd/0xe0 [ 6872.060438] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6872.060445] do_raw_spin_lock+0x177/0x190 [ 6872.060450] _raw_spin_lock+0x21/0x30 [ 6872.060455] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6872.060725] obd_mod_free+0x46/0x140 [ptlrpc] [ 6872.060924] mdc_commit_open+0x91/0xb0 [mdc] [ 6872.060975] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6872.062404] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6872.062630] after_reply+0xe2c/0x1780 [ptlrpc] [ 6872.062872] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6872.063257] ? __next_timer_interrupt+0x160/0x160 [ 6872.063268] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6872.063487] ? do_wait_intr+0xf0/0xf0 [ 6872.063497] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6872.063820] kthread+0x1d1/0x200 [ 6872.063826] ? set_kthread_struct+0x70/0x70 [ 6872.063830] ret_from_fork+0x1f/0x30 [ 6872.070529] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6872.070715] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6872.070722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6872.070725] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6872.070738] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6872.070742] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6872.070753] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6872.070755] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6872.070757] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6872.070759] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6872.070762] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6872.070766] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6872.070769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6872.070771] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6872.070778] Call Trace: [ 6872.070784] [ 6872.070789] ? show_regs.cold.9+0x22/0x2f [ 6872.070812] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6872.070818] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6872.070822] ? watchdog+0x40/0x40 [ 6872.070826] ? hrtimer_interrupt+0x128/0x2c0 [ 6872.070831] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6872.070953] ? apic_timer_interrupt+0xf/0x20 [ 6872.070960] [ 6872.070965] ? native_safe_halt+0x2e/0x50 [ 6872.070970] kvm_wait+0xcd/0xe0 [ 6872.070977] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6872.070984] do_raw_spin_lock+0x177/0x190 [ 6872.070989] _raw_spin_lock+0x21/0x30 [ 6872.070993] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6872.071552] ? do_raw_spin_unlock+0x75/0x190 [ 6872.071558] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6872.071936] ? do_wait_intr+0xf0/0xf0 [ 6872.071945] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6872.073022] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6872.084951] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6872.085042] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6872.085051] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6872.085054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6872.085068] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6872.085072] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6872.085077] RAX: 00000000000055dc RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6872.085079] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6872.085202] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6872.085205] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6872.085207] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6872.085211] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6872.085213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6872.085215] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6872.085223] Call Trace: [ 6872.085229] [ 6872.085235] ? show_regs.cold.9+0x22/0x2f [ 6872.085245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6872.085267] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6872.085349] ? watchdog+0x40/0x40 [ 6872.085356] ? hrtimer_interrupt+0x128/0x2c0 [ 6872.085360] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6872.085369] ? apic_timer_interrupt+0xf/0x20 [ 6872.085379] [ 6872.085381] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6872.085387] do_raw_spin_lock+0x177/0x190 [ 6872.085392] _raw_spin_lock+0x21/0x30 [ 6872.085397] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6872.089857] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6872.090527] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6872.090642] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6872.090681] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6872.090722] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6872.090748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6872.090839] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6872.162118] ? do_raw_spin_unlock+0x75/0x190 [ 6872.162141] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6872.162417] ? do_wait_intr+0xf0/0xf0 [ 6872.162426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6872.162629] mdc_close+0x53b/0xd70 [mdc] [ 6872.162668] lmv_close+0x18d/0x490 [lmv] [ 6872.162702] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6872.162856] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6872.162938] ll_md_close+0x274/0x800 [lustre] [ 6872.163020] ll_file_release+0x128/0xc50 [lustre] [ 6872.163100] __fput+0x10d/0x440 [ 6872.163111] ____fput+0x12/0x20 [ 6872.163116] task_work_run+0xc6/0x110 [ 6872.163123] exit_to_usermode_loop+0x1dd/0x1f0 [ 6872.163130] do_syscall_64+0x3d6/0x440 [ 6872.163134] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6872.163143] RIP: 0033:0x7f1a63983cd8 [ 6872.163150] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6872.163154] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6872.163159] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6872.163162] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6872.163164] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6872.163166] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6872.163168] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6872.913466] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6872.915485] lmv_intent_open+0x338/0x1150 [lmv] [ 6872.917146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6872.921958] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6872.925077] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6872.928849] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6872.933815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6872.938762] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6872.942055] ? do_raw_spin_unlock+0x75/0x190 [ 6872.945220] ? _raw_spin_unlock+0x12/0x30 [ 6872.946925] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6872.948827] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6872.951916] ll_atomic_open+0x36a/0xd70 [lustre] [ 6872.958177] lookup_open+0x356/0xb40 [ 6872.963150] do_last+0x313/0xfc0 [ 6872.964579] ? nd_jump_root+0xe5/0x160 [ 6872.967403] ? path_init+0x437/0x520 [ 6872.970115] path_openat+0xf7/0x500 [ 6872.972679] ? do_raw_spin_unlock+0x75/0x190 [ 6872.976515] do_filp_open+0x99/0x140 [ 6872.978251] ? getname_flags+0x6e/0x330 [ 6872.981176] ? __check_object_size+0xff/0x256 [ 6872.989199] ? do_raw_spin_unlock+0x75/0x190 [ 6872.991484] ? _raw_spin_unlock+0x12/0x30 [ 6872.995838] do_sys_openat2+0x2b4/0x410 [ 6872.998849] do_sys_open+0x73/0xa0 [ 6873.001903] __x64_sys_openat+0x24/0x30 [ 6873.004916] do_syscall_64+0xc1/0x440 [ 6873.009522] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6873.012964] RIP: 0033:0x7f1a63983332 [ 6873.014903] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6873.028064] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6873.034699] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6873.049989] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6873.055875] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6873.068843] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6873.076896] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6876.061806] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6876.079726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6876.177153] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6876.202584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6876.221326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6876.232446] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6876.256259] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6876.270438] RAX: 0000000000003784 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6876.282016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6876.295082] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6876.298488] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6876.315943] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6876.328358] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6876.345698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6876.355940] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6876.365205] Call Trace: [ 6876.372330] [ 6876.373996] ? show_regs.cold.9+0x22/0x2f [ 6876.382956] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6876.390945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6876.403899] ? watchdog+0x40/0x40 [ 6876.405132] ? hrtimer_interrupt+0x128/0x2c0 [ 6876.409634] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6876.414920] ? apic_timer_interrupt+0xf/0x20 [ 6876.418707] [ 6876.421421] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6876.425320] do_raw_spin_lock+0x177/0x190 [ 6876.428125] _raw_spin_lock+0x21/0x30 [ 6876.430531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6876.435207] ? do_raw_spin_unlock+0x75/0x190 [ 6876.440996] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6876.449888] ? do_wait_intr+0xf0/0xf0 [ 6876.454958] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6876.465253] mdc_close+0x53b/0xd70 [mdc] [ 6876.479855] lmv_close+0x18d/0x490 [lmv] [ 6876.490603] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6876.504647] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6876.515185] ll_md_close+0x274/0x800 [lustre] [ 6876.525322] ll_file_release+0x128/0xc50 [lustre] [ 6876.527183] __fput+0x10d/0x440 [ 6876.534107] ____fput+0x12/0x20 [ 6876.540004] task_work_run+0xc6/0x110 [ 6876.552925] exit_to_usermode_loop+0x1dd/0x1f0 [ 6876.555665] do_syscall_64+0x3d6/0x440 [ 6876.557082] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6876.566802] RIP: 0033:0x7f1a63983cd8 [ 6876.570824] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6876.594764] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6876.598798] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6876.604009] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6876.610790] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6876.617751] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6876.620886] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6877.154081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4854s! [ 6877.157028] Showing busy workqueues and worker pools: [ 6877.163175] workqueue events: flags=0x0 [ 6877.166001] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6877.170954] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6877.180829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6877.184628] pending: sec_gc_main [ptlrpc] [ 6877.192836] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6877.197065] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6877.204617] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6877.211199] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6877.221599] workqueue events_long: flags=0x0 [ 6877.226760] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6877.233868] pending: lu_objects_flush [obdclass] [ 6877.237031] workqueue events_unbound: flags=0x2 [ 6877.240507] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6877.249081] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6877.259375] workqueue events_power_efficient: flags=0x80 [ 6877.266913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6877.273661] pending: neigh_periodic_work, do_cache_clean [ 6877.276824] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6877.282952] pending: neigh_periodic_work, check_lifetime [ 6877.286838] workqueue mm_percpu_wq: flags=0x8 [ 6877.292645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6877.298320] pending: vmstat_update [ 6877.301996] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6877.309885] pending: vmstat_update [ 6877.314977] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6877.333206] pending: vmstat_update [ 6877.337432] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6877.350211] pending: vmstat_update [ 6877.351565] workqueue writeback: flags=0x4a [ 6877.371315] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6877.386436] pending: wb_workfn, wb_workfn [ 6877.393808] workqueue xprtiod: flags=0xa [ 6877.403820] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6877.416800] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6877.425373] workqueue ipv6_addrconf: flags=0x40008 [ 6877.431858] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6877.437285] pending: addrconf_verify_work [ 6877.440022] workqueue knbd0-recv: flags=0x1a [ 6877.449067] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6877.465748] in-flight: 647:recv_work [ 6877.469803] workqueue ptlrpc_pinger: flags=0x2 [ 6877.478085] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6877.486892] pending: ptlrpc_pinger_main [ptlrpc] [ 6877.508582] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6867s workers=2 idle: 652 [ 6900.058100] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6900.059127] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 6900.061449] Modules linked in: lustre(O) [ 6900.075175] Modules linked in: [ 6900.075176] osp(O) ofd(O) lod(O) [ 6900.076332] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 6900.077539] mdt(O) [ 6900.080431] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic [ 6900.082612] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6900.082673] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6900.082745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6900.082748] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6900.082761] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6900.082765] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6900.082770] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6900.082773] RDX: 0000000000000e4d RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6900.082775] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6900.082777] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6900.082780] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6900.082783] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6900.082786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6900.082788] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6900.082797] Call Trace: [ 6900.082802] [ 6900.082808] ? show_regs.cold.9+0x22/0x2f [ 6900.082817] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6900.082824] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6900.082828] ? watchdog+0x40/0x40 [ 6900.082833] ? hrtimer_interrupt+0x128/0x2c0 [ 6900.082837] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6900.082846] ? apic_timer_interrupt+0xf/0x20 [ 6900.082851] [ 6900.082854] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6900.082859] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6900.082864] do_raw_spin_lock+0x177/0x190 [ 6900.082870] _raw_spin_lock+0x21/0x30 [ 6900.082875] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6900.083734] ghash_clmulni_intel ata_piix [ 6900.099555] obd_mod_free+0x46/0x140 [ptlrpc] [ 6900.099852] mdc_commit_open+0x91/0xb0 [mdc] [ 6900.099889] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6900.100215] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6900.100572] after_reply+0xe2c/0x1780 [ptlrpc] [ 6900.101211] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6900.127928] serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6900.127953] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6900.127960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6900.127963] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6900.127978] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6900.127982] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6900.127987] RAX: 0000000000007f3f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6900.127990] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6900.127992] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6900.127994] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6900.142867] ? __next_timer_interrupt+0x160/0x160 [ 6900.142889] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6900.143477] ? do_wait_intr+0xf0/0xf0 [ 6900.143488] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6900.143762] kthread+0x1d1/0x200 [ 6900.143768] ? set_kthread_struct+0x70/0x70 [ 6900.143772] ret_from_fork+0x1f/0x30 [ 6900.382364] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6900.389355] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6900.397031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6900.398773] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6900.403634] Call Trace: [ 6900.405432] [ 6900.408589] ? show_regs.cold.9+0x22/0x2f [ 6900.409724] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6900.411500] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6900.414929] ? watchdog+0x40/0x40 [ 6900.416797] ? hrtimer_interrupt+0x128/0x2c0 [ 6900.420299] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6900.422468] ? apic_timer_interrupt+0xf/0x20 [ 6900.424873] [ 6900.425938] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6900.428387] do_raw_spin_lock+0x177/0x190 [ 6900.430301] _raw_spin_lock+0x21/0x30 [ 6900.431603] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6900.433446] ? do_raw_spin_unlock+0x75/0x190 [ 6900.435849] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6900.438644] ? do_wait_intr+0xf0/0xf0 [ 6900.440915] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6900.442910] mdc_close+0x53b/0xd70 [mdc] [ 6900.446305] lmv_close+0x18d/0x490 [lmv] [ 6900.448262] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6900.450669] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6900.452602] ll_md_close+0x274/0x800 [lustre] [ 6900.455979] ll_file_release+0x128/0xc50 [lustre] [ 6900.458250] __fput+0x10d/0x440 [ 6900.459437] ____fput+0x12/0x20 [ 6900.463234] task_work_run+0xc6/0x110 [ 6900.465114] exit_to_usermode_loop+0x1dd/0x1f0 [ 6900.467214] do_syscall_64+0x3d6/0x440 [ 6900.474278] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6900.477632] RIP: 0033:0x7f1a63983cd8 [ 6900.478835] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6900.489074] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6900.492747] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6900.495745] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6900.499858] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6900.511542] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6900.514077] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6904.054264] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6904.061974] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6904.065107] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6904.065258] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6904.065266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6904.065269] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6904.065284] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6904.065289] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6904.065294] RAX: 0000000000007886 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6904.065297] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6904.065299] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6904.065302] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6904.065304] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6904.065307] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6904.065310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6904.065313] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6904.065323] Call Trace: [ 6904.065328] [ 6904.065334] ? show_regs.cold.9+0x22/0x2f [ 6904.065344] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6904.065351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6904.065355] ? watchdog+0x40/0x40 [ 6904.065361] ? hrtimer_interrupt+0x128/0x2c0 [ 6904.065365] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6904.065373] ? apic_timer_interrupt+0xf/0x20 [ 6904.065378] [ 6904.065381] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6904.065387] do_raw_spin_lock+0x177/0x190 [ 6904.065393] _raw_spin_lock+0x21/0x30 [ 6904.065398] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6904.066068] ? do_raw_spin_unlock+0x75/0x190 [ 6904.066074] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6904.066358] ? do_wait_intr+0xf0/0xf0 [ 6904.066367] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6904.066576] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6904.067098] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6904.067398] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6904.067503] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6904.067542] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6904.067568] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6904.067593] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6904.067752] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6904.068294] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6904.068328] lmv_intent_open+0x338/0x1150 [lmv] [ 6904.068361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6904.068450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6904.068528] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6904.068548] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6904.068634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6904.069034] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6904.069129] ? do_raw_spin_unlock+0x75/0x190 [ 6904.069136] ? _raw_spin_unlock+0x12/0x30 [ 6904.069142] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6904.069170] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6904.069260] ll_atomic_open+0x36a/0xd70 [lustre] [ 6904.069355] lookup_open+0x356/0xb40 [ 6904.069365] do_last+0x313/0xfc0 [ 6904.069371] ? nd_jump_root+0xe5/0x160 [ 6904.069376] ? path_init+0x437/0x520 [ 6904.069381] path_openat+0xf7/0x500 [ 6904.069386] ? do_raw_spin_unlock+0x75/0x190 [ 6904.069391] do_filp_open+0x99/0x140 [ 6904.069398] ? getname_flags+0x6e/0x330 [ 6904.069403] ? __check_object_size+0xff/0x256 [ 6904.069410] ? do_raw_spin_unlock+0x75/0x190 [ 6904.069414] ? _raw_spin_unlock+0x12/0x30 [ 6904.069420] do_sys_openat2+0x2b4/0x410 [ 6904.069424] do_sys_open+0x73/0xa0 [ 6904.069429] __x64_sys_openat+0x24/0x30 [ 6904.069433] do_syscall_64+0xc1/0x440 [ 6904.069440] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6904.069446] RIP: 0033:0x7f1a63983332 [ 6904.069453] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6904.069458] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6904.069463] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6904.069466] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6904.069469] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6904.069471] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6904.069474] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6904.646730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6904.709627] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6904.731492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6904.746611] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6904.755588] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6904.778983] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6904.788168] RAX: 000000000000708b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6904.797010] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6904.807731] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6904.818714] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6904.827196] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6904.833346] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6904.848681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6904.854720] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6904.860865] Call Trace: [ 6904.864819] [ 6904.865850] ? show_regs.cold.9+0x22/0x2f [ 6904.869557] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6904.872917] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6904.875855] ? watchdog+0x40/0x40 [ 6904.877859] ? hrtimer_interrupt+0x128/0x2c0 [ 6904.880896] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6904.884197] ? apic_timer_interrupt+0xf/0x20 [ 6904.886731] [ 6904.888591] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6904.891728] do_raw_spin_lock+0x177/0x190 [ 6904.893801] _raw_spin_lock+0x21/0x30 [ 6904.896128] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6904.899043] ? do_raw_spin_unlock+0x75/0x190 [ 6904.901238] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6904.904267] ? do_wait_intr+0xf0/0xf0 [ 6904.906292] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6904.917929] mdc_close+0x53b/0xd70 [mdc] [ 6904.924841] lmv_close+0x18d/0x490 [lmv] [ 6904.928410] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6904.937296] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6904.943650] ll_md_close+0x274/0x800 [lustre] [ 6904.946916] ll_file_release+0x128/0xc50 [lustre] [ 6904.952578] __fput+0x10d/0x440 [ 6904.955690] ____fput+0x12/0x20 [ 6904.961160] task_work_run+0xc6/0x110 [ 6904.962497] exit_to_usermode_loop+0x1dd/0x1f0 [ 6904.975594] do_syscall_64+0x3d6/0x440 [ 6904.980152] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6904.987221] RIP: 0033:0x7f1a63983cd8 [ 6904.995936] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6905.017165] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6905.022637] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6905.026555] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6905.034270] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6905.041545] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6905.050570] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6907.878220] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4884s! [ 6907.884262] Showing busy workqueues and worker pools: [ 6907.886540] workqueue events: flags=0x0 [ 6907.888470] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6907.891574] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6907.894524] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6907.897533] pending: sec_gc_main [ptlrpc] [ 6907.899862] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6907.902886] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6907.915431] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6907.921642] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6907.930505] workqueue events_long: flags=0x0 [ 6907.933060] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6907.938500] pending: lu_objects_flush [obdclass] [ 6907.943316] workqueue events_unbound: flags=0x2 [ 6907.947439] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6907.957250] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6907.965894] workqueue events_power_efficient: flags=0x80 [ 6907.969579] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6907.974578] pending: neigh_periodic_work, do_cache_clean [ 6907.979639] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6907.986051] pending: neigh_periodic_work, check_lifetime [ 6907.992523] workqueue mm_percpu_wq: flags=0x8 [ 6907.997638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6908.003716] pending: vmstat_update [ 6908.008097] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6908.014041] pending: vmstat_update [ 6908.016479] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6908.023558] pending: vmstat_update [ 6908.026689] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6908.034673] pending: vmstat_update [ 6908.040236] workqueue writeback: flags=0x4a [ 6908.044433] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6908.047556] pending: wb_workfn, wb_workfn [ 6908.055786] workqueue xprtiod: flags=0xa [ 6908.061042] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6908.068078] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6908.075511] workqueue ipv6_addrconf: flags=0x40008 [ 6908.078659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6908.083770] pending: addrconf_verify_work [ 6908.088482] workqueue knbd0-recv: flags=0x1a [ 6908.090748] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6908.094552] in-flight: 647:recv_work [ 6908.096327] workqueue ptlrpc_pinger: flags=0x2 [ 6908.097724] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6908.104504] pending: ptlrpc_pinger_main [ptlrpc] [ 6908.110345] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6898s workers=2 idle: 652 [ 6928.056031] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 6928.058029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 6928.059029] Modules linked in: [ 6928.067016] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6928.067099] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6928.067106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6928.067109] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6928.067122] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6928.067126] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6928.067130] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6928.067133] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6928.067134] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6928.067137] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6928.067139] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6928.067142] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6928.067145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6928.067147] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6928.067156] Call Trace: [ 6928.067162] [ 6928.067167] ? show_regs.cold.9+0x22/0x2f [ 6928.067177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6928.067183] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6928.067187] ? watchdog+0x40/0x40 [ 6928.067191] ? hrtimer_interrupt+0x128/0x2c0 [ 6928.067195] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6928.067202] ? apic_timer_interrupt+0xf/0x20 [ 6928.067206] [ 6928.067209] ? native_safe_halt+0x2e/0x50 [ 6928.067213] kvm_wait+0xcd/0xe0 [ 6928.067221] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6928.067228] do_raw_spin_lock+0x177/0x190 [ 6928.067233] _raw_spin_lock+0x21/0x30 [ 6928.067238] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6928.070091] lustre(O) [ 6928.121004] obd_mod_free+0x46/0x140 [ptlrpc] [ 6928.128290] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6928.136427] mdc_commit_open+0x91/0xb0 [mdc] [ 6928.143197] [ 6928.143207] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6928.143215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6928.143218] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6928.143231] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6928.143235] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6928.143239] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6928.143242] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6928.143244] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6928.143246] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6928.143248] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6928.143251] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6928.143254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6928.143256] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6928.143264] Call Trace: [ 6928.143274] [ 6928.143280] ? show_regs.cold.9+0x22/0x2f [ 6928.143289] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6928.143298] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6928.143302] ? watchdog+0x40/0x40 [ 6928.143306] ? hrtimer_interrupt+0x128/0x2c0 [ 6928.143310] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6928.143317] ? apic_timer_interrupt+0xf/0x20 [ 6928.143321] [ 6928.143324] ? native_safe_halt+0x2e/0x50 [ 6928.143328] kvm_wait+0xcd/0xe0 [ 6928.143335] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6928.143344] do_raw_spin_lock+0x177/0x190 [ 6928.143348] _raw_spin_lock+0x21/0x30 [ 6928.143351] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6928.143629] ? do_raw_spin_unlock+0x75/0x190 [ 6928.143635] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6928.143899] ? do_wait_intr+0xf0/0xf0 [ 6928.143907] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6928.158981] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6928.159281] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6928.167388] mdc_close+0x53b/0xd70 [mdc] [ 6928.167437] lmv_close+0x18d/0x490 [lmv] [ 6928.167466] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6928.167575] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6928.167656] ll_md_close+0x274/0x800 [lustre] [ 6928.174186] after_reply+0xe2c/0x1780 [ptlrpc] [ 6928.174460] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6928.174738] ? __next_timer_interrupt+0x160/0x160 [ 6928.174749] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6928.174956] ? do_wait_intr+0xf0/0xf0 [ 6928.174966] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6928.175171] kthread+0x1d1/0x200 [ 6928.175178] ? set_kthread_struct+0x70/0x70 [ 6928.175182] ret_from_fork+0x1f/0x30 [ 6928.573058] ll_file_release+0x128/0xc50 [lustre] [ 6928.576455] __fput+0x10d/0x440 [ 6928.577894] ____fput+0x12/0x20 [ 6928.579131] task_work_run+0xc6/0x110 [ 6928.580629] exit_to_usermode_loop+0x1dd/0x1f0 [ 6928.585914] do_syscall_64+0x3d6/0x440 [ 6928.587253] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6928.594541] RIP: 0033:0x7f1a63983cd8 [ 6928.595897] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6928.607658] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6928.613111] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6928.621343] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6928.626538] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6928.630689] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6928.633190] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6932.057076] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6932.061618] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6932.088972] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6932.095270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6932.100896] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6932.103215] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6932.114225] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6932.117947] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6932.121834] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6932.126284] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6932.130521] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6932.134298] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6932.138370] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6932.143219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6932.147231] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6932.151458] Call Trace: [ 6932.153143] [ 6932.154182] ? show_regs.cold.9+0x22/0x2f [ 6932.156320] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6932.158841] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6932.161897] ? watchdog+0x40/0x40 [ 6932.163598] ? hrtimer_interrupt+0x128/0x2c0 [ 6932.166305] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6932.169468] ? apic_timer_interrupt+0xf/0x20 [ 6932.171782] [ 6932.172992] ? native_safe_halt+0x2e/0x50 [ 6932.175160] kvm_wait+0xcd/0xe0 [ 6932.176636] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6932.179529] do_raw_spin_lock+0x177/0x190 [ 6932.182153] _raw_spin_lock+0x21/0x30 [ 6932.183986] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6932.187552] ? do_raw_spin_unlock+0x75/0x190 [ 6932.190267] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6932.194153] ? do_wait_intr+0xf0/0xf0 [ 6932.196200] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6932.199700] mdc_close+0x53b/0xd70 [mdc] [ 6932.202446] lmv_close+0x18d/0x490 [lmv] [ 6932.205477] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6932.209485] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6932.212919] ll_md_close+0x274/0x800 [lustre] [ 6932.215917] ll_file_release+0x128/0xc50 [lustre] [ 6932.218742] __fput+0x10d/0x440 [ 6932.220217] ____fput+0x12/0x20 [ 6932.221973] task_work_run+0xc6/0x110 [ 6932.224353] exit_to_usermode_loop+0x1dd/0x1f0 [ 6932.232557] do_syscall_64+0x3d6/0x440 [ 6932.235394] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6932.239659] RIP: 0033:0x7f1a63983cd8 [ 6932.244118] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6932.265904] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6932.274112] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6932.282407] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6932.289843] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6932.306904] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6932.316612] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6936.060254] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6936.073105] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6936.115127] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6936.123315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6936.128151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6936.132505] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6936.153353] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6936.157525] RAX: 0000000000007e10 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6936.164510] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6936.168620] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6936.172298] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6936.179256] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6936.184356] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6936.193243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6936.198353] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6936.203544] Call Trace: [ 6936.206088] [ 6936.210149] ? show_regs.cold.9+0x22/0x2f [ 6936.212463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6936.217592] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6936.220111] ? watchdog+0x40/0x40 [ 6936.221627] ? hrtimer_interrupt+0x128/0x2c0 [ 6936.224515] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6936.228329] ? apic_timer_interrupt+0xf/0x20 [ 6936.231124] [ 6936.232168] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6936.236134] do_raw_spin_lock+0x177/0x190 [ 6936.238631] _raw_spin_lock+0x21/0x30 [ 6936.240506] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6936.244404] ? do_raw_spin_unlock+0x75/0x190 [ 6936.246644] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6936.250475] ? do_wait_intr+0xf0/0xf0 [ 6936.253019] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6936.259841] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6936.266124] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6936.274571] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6936.282634] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6936.289812] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6936.292046] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6936.295759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6936.298602] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6936.303069] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6936.305445] lmv_intent_open+0x338/0x1150 [lmv] [ 6936.307515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6936.310411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6936.314461] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6936.317424] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6936.320828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6936.324340] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6936.327502] ? do_raw_spin_unlock+0x75/0x190 [ 6936.330563] ? _raw_spin_unlock+0x12/0x30 [ 6936.332705] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6936.334762] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6936.338086] ll_atomic_open+0x36a/0xd70 [lustre] [ 6936.343194] lookup_open+0x356/0xb40 [ 6936.345490] do_last+0x313/0xfc0 [ 6936.347269] ? nd_jump_root+0xe5/0x160 [ 6936.350418] ? path_init+0x437/0x520 [ 6936.354423] path_openat+0xf7/0x500 [ 6936.358381] ? do_raw_spin_unlock+0x75/0x190 [ 6936.364276] do_filp_open+0x99/0x140 [ 6936.368130] ? getname_flags+0x6e/0x330 [ 6936.371404] ? __check_object_size+0xff/0x256 [ 6936.377442] ? do_raw_spin_unlock+0x75/0x190 [ 6936.384551] ? _raw_spin_unlock+0x12/0x30 [ 6936.388411] do_sys_openat2+0x2b4/0x410 [ 6936.395577] do_sys_open+0x73/0xa0 [ 6936.409295] __x64_sys_openat+0x24/0x30 [ 6936.418116] do_syscall_64+0xc1/0x440 [ 6936.423434] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6936.432252] RIP: 0033:0x7f1a63983332 [ 6936.434509] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6936.448748] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6936.452535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6936.456069] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6936.460514] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6936.464714] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6936.468918] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6938.596063] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4915s! [ 6938.599818] Showing busy workqueues and worker pools: [ 6938.602294] workqueue events: flags=0x0 [ 6938.604205] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6938.607395] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6938.610994] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6938.619996] pending: sec_gc_main [ptlrpc] [ 6938.622338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6938.630762] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6938.634761] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6938.638918] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6938.643110] workqueue events_long: flags=0x0 [ 6938.645146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6938.648073] pending: lu_objects_flush [obdclass] [ 6938.650822] workqueue events_unbound: flags=0x2 [ 6938.655787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6938.660171] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6938.666883] workqueue events_power_efficient: flags=0x80 [ 6938.671916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6938.674918] pending: neigh_periodic_work, do_cache_clean [ 6938.678313] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6938.694357] pending: neigh_periodic_work, check_lifetime [ 6938.698904] workqueue mm_percpu_wq: flags=0x8 [ 6938.702044] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6938.705242] pending: vmstat_update [ 6938.706302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6938.710029] pending: vmstat_update [ 6938.712412] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6938.716363] pending: vmstat_update [ 6938.718515] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6938.722065] pending: vmstat_update [ 6938.723985] workqueue writeback: flags=0x4a [ 6938.726405] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6938.729819] pending: wb_workfn, wb_workfn [ 6938.732263] workqueue xprtiod: flags=0xa [ 6938.733594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6938.736758] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6938.740530] workqueue ipv6_addrconf: flags=0x40008 [ 6938.742874] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6938.745987] pending: addrconf_verify_work [ 6938.748657] workqueue knbd0-recv: flags=0x1a [ 6938.750113] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6938.752386] in-flight: 647:recv_work [ 6938.753785] workqueue ptlrpc_pinger: flags=0x2 [ 6938.755294] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6938.757248] pending: ptlrpc_pinger_main [ptlrpc] [ 6938.759250] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6928s workers=2 idle: 652 [ 6942.520319] rcu: INFO: rcu_sched self-detected stall on CPU [ 6942.525927] rcu: 3-...!: (3049322 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 6942.537059] (t=4920131 jiffies g=269633 q=1865) [ 6942.540090] rcu: rcu_sched kthread starved for 4919779 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 6942.551875] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6942.556550] rcu: RCU grace-period kthread stack dump: [ 6942.559027] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6942.566715] Call Trace: [ 6942.568390] __schedule+0x351/0xcb0 [ 6942.572323] schedule+0xc0/0x180 [ 6942.574243] schedule_timeout+0xb4/0x190 [ 6942.575933] ? do_raw_spin_unlock+0x75/0x190 [ 6942.577522] ? __next_timer_interrupt+0x160/0x160 [ 6942.579551] ? prepare_to_swait_event+0xf3/0x140 [ 6942.582713] rcu_gp_kthread+0x8dd/0x15b0 [ 6942.585898] ? force_qs_rnp+0x2c0/0x2c0 [ 6942.587847] kthread+0x1d1/0x200 [ 6942.589671] ? set_kthread_struct+0x70/0x70 [ 6942.591152] ret_from_fork+0x1f/0x30 [ 6942.592376] rcu: Stack dump where RCU GP kthread last ran: [ 6942.601921] Sending NMI from CPU 3 to CPUs 2: [ 6942.603854] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 6942.604819] NMI backtrace for cpu 3 [ 6942.607479] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6942.620537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6942.631095] Call Trace: [ 6942.632161] [ 6942.635067] ? dump_stack+0xbb/0x10e [ 6942.637717] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6942.642561] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6942.649836] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6942.652256] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6942.658237] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6942.660555] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6942.664170] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6942.671066] ? update_process_times+0x6d/0xc0 [ 6942.676147] ? tick_sched_handle.isra.18+0x27/0x90 [ 6942.678653] ? tick_sched_timer+0x53/0xb0 [ 6942.685368] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6942.686859] ? tick_sched_handle.isra.18+0x90/0x90 [ 6942.690158] ? hrtimer_interrupt+0x128/0x2c0 [ 6942.692357] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6942.697847] ? apic_timer_interrupt+0xf/0x20 [ 6942.699610] [ 6942.700499] ? native_safe_halt+0x2e/0x50 [ 6942.703103] ? kvm_wait+0xcd/0xe0 [ 6942.706166] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6942.708305] ? do_raw_spin_lock+0x177/0x190 [ 6942.714093] ? _raw_spin_lock+0x21/0x30 [ 6942.717315] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6942.722728] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6942.727291] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6942.733876] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6942.740407] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6942.747521] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6942.756421] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6942.759466] ? __next_timer_interrupt+0x160/0x160 [ 6942.764259] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6942.769499] ? do_wait_intr+0xf0/0xf0 [ 6942.774902] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6942.782060] ? kthread+0x1d1/0x200 [ 6942.789216] ? set_kthread_struct+0x70/0x70 [ 6942.796098] ? ret_from_fork+0x1f/0x30 [ 6956.056061] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 6956.059659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6956.081363] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6956.085988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6956.089697] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6956.091319] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6956.097212] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6956.099818] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6956.102812] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6956.105236] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6956.107649] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6956.110103] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6956.112644] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6956.115523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6956.117307] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6956.119625] Call Trace: [ 6956.120435] [ 6956.121106] ? show_regs.cold.9+0x22/0x2f [ 6956.122339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6956.123924] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6956.125434] ? watchdog+0x40/0x40 [ 6956.126798] ? hrtimer_interrupt+0x128/0x2c0 [ 6956.128427] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6956.129978] ? apic_timer_interrupt+0xf/0x20 [ 6956.131555] [ 6956.132205] ? native_safe_halt+0x2e/0x50 [ 6956.133660] kvm_wait+0xcd/0xe0 [ 6956.134872] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6956.136586] do_raw_spin_lock+0x177/0x190 [ 6956.138211] _raw_spin_lock+0x21/0x30 [ 6956.139379] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6956.141518] ? do_raw_spin_unlock+0x75/0x190 [ 6956.142883] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6956.145088] ? do_wait_intr+0xf0/0xf0 [ 6956.146549] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6956.148780] mdc_close+0x53b/0xd70 [mdc] [ 6956.150757] lmv_close+0x18d/0x490 [lmv] [ 6956.152065] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6956.153990] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6956.155838] ll_md_close+0x274/0x800 [lustre] [ 6956.157586] ll_file_release+0x128/0xc50 [lustre] [ 6956.159439] __fput+0x10d/0x440 [ 6956.160688] ____fput+0x12/0x20 [ 6956.161805] task_work_run+0xc6/0x110 [ 6956.163229] exit_to_usermode_loop+0x1dd/0x1f0 [ 6956.164684] do_syscall_64+0x3d6/0x440 [ 6956.165990] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6956.167827] RIP: 0033:0x7f1a63983cd8 [ 6956.169203] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6956.176094] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6956.178988] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6956.181521] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6956.184117] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6956.186578] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6956.188925] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6964.055007] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6964.058488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6964.086775] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6964.092715] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6964.104299] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6964.111902] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6964.136753] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6964.145770] RAX: 0000000000006c0d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6964.157330] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6964.165958] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6964.177115] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6964.187792] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6964.194795] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6964.206903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6964.210777] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6964.215766] Call Trace: [ 6964.217104] [ 6964.218741] ? show_regs.cold.9+0x22/0x2f [ 6964.221152] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6964.223663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6964.226187] ? watchdog+0x40/0x40 [ 6964.228165] ? hrtimer_interrupt+0x128/0x2c0 [ 6964.231173] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6964.238223] ? apic_timer_interrupt+0xf/0x20 [ 6964.239943] [ 6964.240706] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6964.245340] do_raw_spin_lock+0x177/0x190 [ 6964.248121] _raw_spin_lock+0x21/0x30 [ 6964.252357] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6964.256256] ? do_raw_spin_unlock+0x75/0x190 [ 6964.257890] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6964.267074] ? do_wait_intr+0xf0/0xf0 [ 6964.273907] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6964.277417] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6964.279668] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6964.282066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6964.287107] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6964.290217] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6964.292336] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6964.296016] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6964.302738] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6964.308665] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6964.314137] lmv_intent_open+0x338/0x1150 [lmv] [ 6964.319743] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6964.324335] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6964.328560] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6964.331218] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6964.333231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6964.336165] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6964.339636] ? do_raw_spin_unlock+0x75/0x190 [ 6964.342059] ? _raw_spin_unlock+0x12/0x30 [ 6964.344328] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6964.346975] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6964.348851] ll_atomic_open+0x36a/0xd70 [lustre] [ 6964.351279] lookup_open+0x356/0xb40 [ 6964.353198] do_last+0x313/0xfc0 [ 6964.356177] ? nd_jump_root+0xe5/0x160 [ 6964.360764] ? path_init+0x437/0x520 [ 6964.365330] path_openat+0xf7/0x500 [ 6964.369245] ? do_raw_spin_unlock+0x75/0x190 [ 6964.378757] do_filp_open+0x99/0x140 [ 6964.381027] ? getname_flags+0x6e/0x330 [ 6964.387607] ? __check_object_size+0xff/0x256 [ 6964.392666] ? do_raw_spin_unlock+0x75/0x190 [ 6964.399304] ? _raw_spin_unlock+0x12/0x30 [ 6964.400940] do_sys_openat2+0x2b4/0x410 [ 6964.405200] do_sys_open+0x73/0xa0 [ 6964.410218] __x64_sys_openat+0x24/0x30 [ 6964.415502] do_syscall_64+0xc1/0x440 [ 6964.422653] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6964.424694] RIP: 0033:0x7f1a63983332 [ 6964.425918] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6964.436091] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6964.438589] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6964.441837] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6964.444974] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6964.447452] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6964.450183] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6968.057136] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6968.058410] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6968.060520] Modules linked in: [ 6968.063363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6968.063427] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6968.063432] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6968.063434] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6968.063444] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6968.063448] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6968.063451] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6968.063453] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6968.063455] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6968.063456] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6968.063457] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6968.063460] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6968.063462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6968.063464] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6968.063473] Call Trace: [ 6968.063478] [ 6968.063483] ? show_regs.cold.9+0x22/0x2f [ 6968.063498] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6968.063505] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6968.063509] ? watchdog+0x40/0x40 [ 6968.063513] ? hrtimer_interrupt+0x128/0x2c0 [ 6968.063517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6968.063523] ? apic_timer_interrupt+0xf/0x20 [ 6968.063527] [ 6968.063530] ? native_safe_halt+0x2e/0x50 [ 6968.063534] kvm_wait+0xcd/0xe0 [ 6968.063540] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6968.063547] do_raw_spin_lock+0x177/0x190 [ 6968.063552] _raw_spin_lock+0x21/0x30 [ 6968.063556] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6968.063900] obd_mod_free+0x46/0x140 [ptlrpc] [ 6968.064101] mdc_commit_open+0x91/0xb0 [mdc] [ 6968.064138] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6968.064300] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6968.065197] after_reply+0xe2c/0x1780 [ptlrpc] [ 6968.065423] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6968.065717] ? __next_timer_interrupt+0x160/0x160 [ 6968.065726] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6968.065989] ? do_wait_intr+0xf0/0xf0 [ 6968.065999] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6968.066172] kthread+0x1d1/0x200 [ 6968.066176] ? set_kthread_struct+0x70/0x70 [ 6968.066178] ret_from_fork+0x1f/0x30 [ 6968.174318] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6968.189239] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6968.193132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6968.196653] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6968.198900] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6968.205502] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6968.208455] RAX: 00000000000045e0 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6968.210791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6968.213731] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6968.216504] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6968.218941] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6968.221473] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6968.224171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6968.226872] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6968.229808] Call Trace: [ 6968.230696] [ 6968.231464] ? show_regs.cold.9+0x22/0x2f [ 6968.233066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6968.234854] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6968.236376] ? watchdog+0x40/0x40 [ 6968.237556] ? hrtimer_interrupt+0x128/0x2c0 [ 6968.239159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6968.240907] ? apic_timer_interrupt+0xf/0x20 [ 6968.242744] [ 6968.244043] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6968.246735] do_raw_spin_lock+0x177/0x190 [ 6968.247958] _raw_spin_lock+0x21/0x30 [ 6968.250106] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6968.252652] ? do_raw_spin_unlock+0x75/0x190 [ 6968.253890] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6968.257985] ? do_wait_intr+0xf0/0xf0 [ 6968.259355] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6968.262161] mdc_close+0x53b/0xd70 [mdc] [ 6968.263533] lmv_close+0x18d/0x490 [lmv] [ 6968.264914] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6968.267079] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6968.268752] ll_md_close+0x274/0x800 [lustre] [ 6968.270154] ll_file_release+0x128/0xc50 [lustre] [ 6968.272009] __fput+0x10d/0x440 [ 6968.273226] ____fput+0x12/0x20 [ 6968.274307] task_work_run+0xc6/0x110 [ 6968.276095] exit_to_usermode_loop+0x1dd/0x1f0 [ 6968.277586] do_syscall_64+0x3d6/0x440 [ 6968.279630] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6968.281156] RIP: 0033:0x7f1a63983cd8 [ 6968.282144] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6968.288673] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6968.291942] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6968.294265] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6968.297127] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6968.299823] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6968.302075] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6968.800671] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4945s! [ 6968.816095] Showing busy workqueues and worker pools: [ 6968.825200] workqueue events: flags=0x0 [ 6968.831081] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6968.833507] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6968.839060] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6968.842805] pending: sec_gc_main [ptlrpc] [ 6968.846713] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6968.852071] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6968.863546] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6968.867092] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6968.873409] workqueue events_long: flags=0x0 [ 6968.875701] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6968.884716] pending: lu_objects_flush [obdclass] [ 6968.888021] workqueue events_unbound: flags=0x2 [ 6968.890816] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6968.893870] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6968.899298] workqueue events_power_efficient: flags=0x80 [ 6968.902957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6968.908471] pending: neigh_periodic_work, do_cache_clean [ 6968.912928] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6968.918487] pending: neigh_periodic_work, check_lifetime [ 6968.927673] workqueue mm_percpu_wq: flags=0x8 [ 6968.931426] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6968.939872] pending: vmstat_update [ 6968.942080] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6968.945527] pending: vmstat_update [ 6968.948809] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6968.954698] pending: vmstat_update [ 6968.956526] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6968.959975] pending: vmstat_update [ 6968.961878] workqueue writeback: flags=0x4a [ 6968.963791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6968.966441] pending: wb_workfn, wb_workfn [ 6968.968746] workqueue xprtiod: flags=0xa [ 6968.971495] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6968.974712] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6968.977634] workqueue ipv6_addrconf: flags=0x40008 [ 6968.982508] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6968.987468] pending: addrconf_verify_work [ 6968.990401] workqueue knbd0-recv: flags=0x1a [ 6968.992715] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6968.999012] in-flight: 647:recv_work [ 6969.001993] workqueue ptlrpc_pinger: flags=0x2 [ 6969.005039] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6969.009476] pending: ptlrpc_pinger_main [ptlrpc] [ 6969.013392] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6959s workers=2 idle: 652 [ 6984.058223] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 6984.068039] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6984.103512] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6984.110428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6984.115601] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6984.120672] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6984.133073] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6984.139032] RAX: 0000000000003285 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6984.147871] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 6984.152651] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6984.160625] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 6984.167665] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 6984.177353] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 6984.185030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6984.194421] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 6984.201499] Call Trace: [ 6984.206456] [ 6984.207528] ? show_regs.cold.9+0x22/0x2f [ 6984.214925] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6984.220144] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6984.227904] ? watchdog+0x40/0x40 [ 6984.233025] ? hrtimer_interrupt+0x128/0x2c0 [ 6984.236478] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6984.241564] ? apic_timer_interrupt+0xf/0x20 [ 6984.243132] [ 6984.243920] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6984.253080] do_raw_spin_lock+0x177/0x190 [ 6984.260029] _raw_spin_lock+0x21/0x30 [ 6984.263007] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6984.269125] ? do_raw_spin_unlock+0x75/0x190 [ 6984.275811] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6984.282997] ? do_wait_intr+0xf0/0xf0 [ 6984.287927] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6984.295613] mdc_close+0x53b/0xd70 [mdc] [ 6984.301461] lmv_close+0x18d/0x490 [lmv] [ 6984.305518] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6984.316898] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6984.322516] ll_md_close+0x274/0x800 [lustre] [ 6984.324497] ll_file_release+0x128/0xc50 [lustre] [ 6984.331019] __fput+0x10d/0x440 [ 6984.334960] ____fput+0x12/0x20 [ 6984.337039] task_work_run+0xc6/0x110 [ 6984.338794] exit_to_usermode_loop+0x1dd/0x1f0 [ 6984.343912] do_syscall_64+0x3d6/0x440 [ 6984.345953] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6984.350987] RIP: 0033:0x7f1a63983cd8 [ 6984.352979] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6984.362396] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6984.369012] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 6984.373523] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6984.379308] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 6984.381531] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 6984.384317] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6996.053678] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 6996.057017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 6996.058458] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 6996.058469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6996.058546] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6996.058551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6996.058554] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6996.058573] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6996.058577] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6996.058581] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 6996.058583] RDX: 000000000000460e RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 6996.058585] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 6996.058588] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 6996.058590] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6996.058593] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 6996.058596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6996.058599] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 6996.058607] Call Trace: [ 6996.058611] [ 6996.058617] ? show_regs.cold.9+0x22/0x2f [ 6996.058625] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6996.058631] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6996.058635] ? watchdog+0x40/0x40 [ 6996.058639] ? hrtimer_interrupt+0x128/0x2c0 [ 6996.058648] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6996.058655] ? apic_timer_interrupt+0xf/0x20 [ 6996.058659] [ 6996.058662] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6996.058667] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6996.058674] do_raw_spin_lock+0x177/0x190 [ 6996.058679] _raw_spin_lock+0x21/0x30 [ 6996.058684] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6996.059172] obd_mod_free+0x46/0x140 [ptlrpc] [ 6996.059449] mdc_commit_open+0x91/0xb0 [mdc] [ 6996.059485] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6996.059678] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6996.059883] after_reply+0xe2c/0x1780 [ptlrpc] [ 6996.060627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6996.060697] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6996.060703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6996.060705] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6996.060714] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6996.060719] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6996.060724] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6996.060726] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 6996.060729] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 6996.060731] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 6996.060734] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 6996.060738] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 6996.060748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6996.060751] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 6996.060760] Call Trace: [ 6996.060767] [ 6996.060770] ? show_regs.cold.9+0x22/0x2f [ 6996.060783] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6996.060795] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6996.060799] ? watchdog+0x40/0x40 [ 6996.060804] ? hrtimer_interrupt+0x128/0x2c0 [ 6996.060814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6996.060827] ? apic_timer_interrupt+0xf/0x20 [ 6996.060836] [ 6996.060843] ? native_safe_halt+0x2e/0x50 [ 6996.060851] kvm_wait+0xcd/0xe0 [ 6996.060865] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6996.060876] do_raw_spin_lock+0x177/0x190 [ 6996.060880] _raw_spin_lock+0x21/0x30 [ 6996.060889] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6996.060254] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6996.061570] ? __next_timer_interrupt+0x160/0x160 [ 6996.061579] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6996.061806] ? do_wait_intr+0xf0/0xf0 [ 6996.061814] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6996.062165] kthread+0x1d1/0x200 [ 6996.062170] ? set_kthread_struct+0x70/0x70 [ 6996.062174] ret_from_fork+0x1f/0x30 [ 6996.064786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6996.064863] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6996.064870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6996.064873] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6996.064884] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6996.064889] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6996.065012] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 6996.065015] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 6996.065017] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 6996.065019] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 6996.065022] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 6996.065025] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 6996.065028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6996.065031] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 6996.065040] Call Trace: [ 6996.065045] [ 6996.065050] ? show_regs.cold.9+0x22/0x2f [ 6996.069249] ? do_raw_spin_unlock+0x75/0x190 [ 6996.069269] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6996.095259] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6996.095286] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6996.095291] ? watchdog+0x40/0x40 [ 6996.099898] ? do_wait_intr+0xf0/0xf0 [ 6996.099918] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6996.107315] ? hrtimer_interrupt+0x128/0x2c0 [ 6996.107332] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6996.107342] ? apic_timer_interrupt+0xf/0x20 [ 6996.107347] [ 6996.107351] ? native_safe_halt+0x2e/0x50 [ 6996.107356] kvm_wait+0xcd/0xe0 [ 6996.107366] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6996.107380] do_raw_spin_lock+0x177/0x190 [ 6996.107385] _raw_spin_lock+0x21/0x30 [ 6996.107389] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6996.107661] ? do_raw_spin_unlock+0x75/0x190 [ 6996.107666] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6996.107931] ? do_wait_intr+0xf0/0xf0 [ 6996.107939] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6996.108237] mdc_close+0x53b/0xd70 [mdc] [ 6996.108283] lmv_close+0x18d/0x490 [lmv] [ 6996.108404] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6996.108516] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6996.108602] ll_md_close+0x274/0x800 [lustre] [ 6996.108696] ll_file_release+0x128/0xc50 [lustre] [ 6996.108783] __fput+0x10d/0x440 [ 6996.108790] ____fput+0x12/0x20 [ 6996.108794] task_work_run+0xc6/0x110 [ 6996.108801] exit_to_usermode_loop+0x1dd/0x1f0 [ 6996.108808] do_syscall_64+0x3d6/0x440 [ 6996.108815] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6996.108821] RIP: 0033:0x7f1a63983cd8 [ 6996.108836] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6996.108840] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6996.108844] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 6996.108847] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 6996.108849] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 6996.108851] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 6996.108854] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6996.487297] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6996.490110] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6996.492425] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6996.495782] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6996.502128] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6996.503967] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6996.505779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6996.507731] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6996.510391] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6996.512318] lmv_intent_open+0x338/0x1150 [lmv] [ 6996.513797] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6996.515876] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6996.517969] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6996.519628] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6996.521577] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6996.523679] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6996.525470] ? do_raw_spin_unlock+0x75/0x190 [ 6996.528902] ? _raw_spin_unlock+0x12/0x30 [ 6996.530301] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6996.535255] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6996.537138] ll_atomic_open+0x36a/0xd70 [lustre] [ 6996.542121] lookup_open+0x356/0xb40 [ 6996.543776] do_last+0x313/0xfc0 [ 6996.544691] ? nd_jump_root+0xe5/0x160 [ 6996.545650] ? path_init+0x437/0x520 [ 6996.546593] path_openat+0xf7/0x500 [ 6996.553547] ? do_raw_spin_unlock+0x75/0x190 [ 6996.555523] do_filp_open+0x99/0x140 [ 6996.556497] ? getname_flags+0x6e/0x330 [ 6996.557599] ? __check_object_size+0xff/0x256 [ 6996.558950] ? do_raw_spin_unlock+0x75/0x190 [ 6996.560473] ? _raw_spin_unlock+0x12/0x30 [ 6996.562954] do_sys_openat2+0x2b4/0x410 [ 6996.564473] do_sys_open+0x73/0xa0 [ 6996.569436] __x64_sys_openat+0x24/0x30 [ 6996.571518] do_syscall_64+0xc1/0x440 [ 6996.577093] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6996.579352] RIP: 0033:0x7f1a63983332 [ 6996.583703] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6996.592773] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6996.595658] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 6996.598593] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 6996.600915] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 6996.606131] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 6996.608589] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 6999.522137] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4976s! [ 6999.525755] Showing busy workqueues and worker pools: [ 6999.527347] workqueue events: flags=0x0 [ 6999.528493] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6999.530315] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 6999.532564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6999.535877] pending: sec_gc_main [ptlrpc] [ 6999.538291] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6999.547191] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 6999.555185] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6999.558568] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 6999.570591] workqueue events_long: flags=0x0 [ 6999.573442] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6999.580693] pending: lu_objects_flush [obdclass] [ 6999.584390] workqueue events_unbound: flags=0x2 [ 6999.587337] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6999.591394] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6999.597085] workqueue events_power_efficient: flags=0x80 [ 6999.602018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6999.605153] pending: neigh_periodic_work, do_cache_clean [ 6999.609243] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6999.618490] pending: neigh_periodic_work, check_lifetime [ 6999.627910] workqueue mm_percpu_wq: flags=0x8 [ 6999.633608] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6999.641492] pending: vmstat_update [ 6999.647387] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6999.655492] pending: vmstat_update [ 6999.659570] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6999.664477] pending: vmstat_update [ 6999.670424] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6999.678382] pending: vmstat_update [ 6999.679841] workqueue writeback: flags=0x4a [ 6999.685481] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6999.688566] pending: wb_workfn, wb_workfn [ 6999.692346] workqueue xprtiod: flags=0xa [ 6999.693938] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6999.700376] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 6999.708613] workqueue ipv6_addrconf: flags=0x40008 [ 6999.710358] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6999.716488] pending: addrconf_verify_work [ 6999.721172] workqueue knbd0-recv: flags=0x1a [ 6999.725353] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6999.730134] in-flight: 647:recv_work [ 6999.731728] workqueue ptlrpc_pinger: flags=0x2 [ 6999.739628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6999.747766] pending: ptlrpc_pinger_main [ptlrpc] [ 6999.753829] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6989s workers=2 idle: 652 [ 7012.056999] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7012.066345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7012.107892] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7012.118806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7012.130232] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7012.133469] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7012.148565] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7012.155667] RAX: 00000000000019aa RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7012.162807] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7012.169462] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7012.173715] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7012.182425] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7012.191254] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7012.199532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7012.202392] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7012.207838] Call Trace: [ 7012.213098] [ 7012.213935] ? show_regs.cold.9+0x22/0x2f [ 7012.220188] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7012.223824] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7012.225166] ? watchdog+0x40/0x40 [ 7012.226676] ? hrtimer_interrupt+0x128/0x2c0 [ 7012.236068] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7012.242781] ? apic_timer_interrupt+0xf/0x20 [ 7012.250753] [ 7012.251498] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7012.257377] do_raw_spin_lock+0x177/0x190 [ 7012.264753] _raw_spin_lock+0x21/0x30 [ 7012.271868] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7012.279421] ? do_raw_spin_unlock+0x75/0x190 [ 7012.285567] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7012.295712] ? do_wait_intr+0xf0/0xf0 [ 7012.298532] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7012.314179] mdc_close+0x53b/0xd70 [mdc] [ 7012.319208] lmv_close+0x18d/0x490 [lmv] [ 7012.323170] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7012.325747] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7012.328164] ll_md_close+0x274/0x800 [lustre] [ 7012.330510] ll_file_release+0x128/0xc50 [lustre] [ 7012.333205] __fput+0x10d/0x440 [ 7012.336442] ____fput+0x12/0x20 [ 7012.337669] task_work_run+0xc6/0x110 [ 7012.339532] exit_to_usermode_loop+0x1dd/0x1f0 [ 7012.342308] do_syscall_64+0x3d6/0x440 [ 7012.344771] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7012.347737] RIP: 0033:0x7f1a63983cd8 [ 7012.350240] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7012.362433] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7012.366530] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7012.370298] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7012.373681] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7012.377557] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7012.380777] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7024.053346] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7024.057060] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7024.058929] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7024.058941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7024.059016] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7024.059021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7024.059437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7024.059490] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7024.059495] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7024.059500] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7024.059502] RDX: 0000000000000da8 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7024.059504] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7024.059506] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7024.059509] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7024.059512] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7024.059515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7024.059517] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7024.059525] Call Trace: [ 7024.059531] [ 7024.059536] ? show_regs.cold.9+0x22/0x2f [ 7024.059544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7024.059550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7024.059554] ? watchdog+0x40/0x40 [ 7024.059558] ? hrtimer_interrupt+0x128/0x2c0 [ 7024.059562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7024.059570] ? apic_timer_interrupt+0xf/0x20 [ 7024.059574] [ 7024.059577] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7024.059582] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7024.059587] do_raw_spin_lock+0x177/0x190 [ 7024.059591] _raw_spin_lock+0x21/0x30 [ 7024.059596] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7024.060963] obd_mod_free+0x46/0x140 [ptlrpc] [ 7024.061953] mdc_commit_open+0x91/0xb0 [mdc] [ 7024.061991] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7024.063564] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7024.064923] after_reply+0xe2c/0x1780 [ptlrpc] [ 7024.066001] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7024.066284] ? __next_timer_interrupt+0x160/0x160 [ 7024.066297] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7024.066506] ? do_wait_intr+0xf0/0xf0 [ 7024.066515] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7024.066851] kthread+0x1d1/0x200 [ 7024.066857] ? set_kthread_struct+0x70/0x70 [ 7024.066861] ret_from_fork+0x1f/0x30 [ 7024.067274] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7024.067435] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7024.067441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7024.067443] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7024.067454] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7024.067458] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7024.067462] RAX: 0000000000004536 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7024.067464] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7024.067466] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7024.067469] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7024.067475] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7024.067478] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7024.067481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7024.067484] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7024.067491] Call Trace: [ 7024.067495] [ 7024.067498] ? show_regs.cold.9+0x22/0x2f [ 7024.067506] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7024.067512] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7024.067516] ? watchdog+0x40/0x40 [ 7024.067520] ? hrtimer_interrupt+0x128/0x2c0 [ 7024.067525] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7024.067531] ? apic_timer_interrupt+0xf/0x20 [ 7024.067535] [ 7024.067537] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7024.067543] do_raw_spin_lock+0x177/0x190 [ 7024.067547] _raw_spin_lock+0x21/0x30 [ 7024.067551] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7024.071378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7024.071452] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7024.071460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7024.071462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7024.071475] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7024.071479] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7024.071484] RAX: 00000000000037d9 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7024.071487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7024.071489] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7024.071492] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7024.071494] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7024.071497] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7024.071500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7024.071502] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7024.071511] Call Trace: [ 7024.071517] [ 7024.071521] ? show_regs.cold.9+0x22/0x2f [ 7024.071531] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7024.071538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7024.071542] ? watchdog+0x40/0x40 [ 7024.071546] ? hrtimer_interrupt+0x128/0x2c0 [ 7024.071550] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7024.071557] ? apic_timer_interrupt+0xf/0x20 [ 7024.071562] [ 7024.071565] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7024.071571] do_raw_spin_lock+0x177/0x190 [ 7024.071576] _raw_spin_lock+0x21/0x30 [ 7024.071580] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7024.072665] ? do_raw_spin_unlock+0x75/0x190 [ 7024.072673] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7024.073631] ? do_wait_intr+0xf0/0xf0 [ 7024.073641] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7024.081688] ? do_raw_spin_unlock+0x75/0x190 [ 7024.081828] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7024.121235] mdc_close+0x53b/0xd70 [mdc] [ 7024.127227] ? do_wait_intr+0xf0/0xf0 [ 7024.127248] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7024.137126] lmv_close+0x18d/0x490 [lmv] [ 7024.137293] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7024.137437] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7024.137532] ll_md_close+0x274/0x800 [lustre] [ 7024.137623] ll_file_release+0x128/0xc50 [lustre] [ 7024.137723] __fput+0x10d/0x440 [ 7024.137732] ____fput+0x12/0x20 [ 7024.137736] task_work_run+0xc6/0x110 [ 7024.137744] exit_to_usermode_loop+0x1dd/0x1f0 [ 7024.137751] do_syscall_64+0x3d6/0x440 [ 7024.137755] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7024.137768] RIP: 0033:0x7f1a63983cd8 [ 7024.137776] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7024.137779] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7024.137784] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7024.137786] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7024.137788] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7024.137790] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7024.137792] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7024.739134] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7024.750848] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7024.760459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7024.764184] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7024.771963] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7024.781198] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7024.789479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7024.797210] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7024.801579] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7024.810280] lmv_intent_open+0x338/0x1150 [lmv] [ 7024.815169] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7024.819116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7024.827767] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7024.835139] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7024.838284] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7024.854534] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7024.859422] ? do_raw_spin_unlock+0x75/0x190 [ 7024.861973] ? _raw_spin_unlock+0x12/0x30 [ 7024.869473] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7024.877990] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7024.886625] ll_atomic_open+0x36a/0xd70 [lustre] [ 7024.894162] lookup_open+0x356/0xb40 [ 7024.902625] do_last+0x313/0xfc0 [ 7024.909332] ? nd_jump_root+0xe5/0x160 [ 7024.913928] ? path_init+0x437/0x520 [ 7024.921940] path_openat+0xf7/0x500 [ 7024.926363] ? do_raw_spin_unlock+0x75/0x190 [ 7024.933467] do_filp_open+0x99/0x140 [ 7024.936223] ? getname_flags+0x6e/0x330 [ 7024.949867] ? __check_object_size+0xff/0x256 [ 7024.957710] ? do_raw_spin_unlock+0x75/0x190 [ 7024.961253] ? _raw_spin_unlock+0x12/0x30 [ 7024.965988] do_sys_openat2+0x2b4/0x410 [ 7024.974164] do_sys_open+0x73/0xa0 [ 7024.975379] __x64_sys_openat+0x24/0x30 [ 7024.981269] do_syscall_64+0xc1/0x440 [ 7024.982708] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7024.994602] RIP: 0033:0x7f1a63983332 [ 7024.999728] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7025.025505] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7025.034297] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7025.043002] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7025.052749] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7025.063450] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7025.070772] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7030.249809] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5007s! [ 7030.261183] Showing busy workqueues and worker pools: [ 7030.266085] workqueue events: flags=0x0 [ 7030.273814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7030.289377] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7030.295054] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7030.302253] pending: sec_gc_main [ptlrpc] [ 7030.308499] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7030.316585] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7030.333346] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7030.339859] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7030.353501] workqueue events_long: flags=0x0 [ 7030.360891] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7030.372392] pending: lu_objects_flush [obdclass] [ 7030.379807] workqueue events_unbound: flags=0x2 [ 7030.392319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7030.412949] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7030.440064] workqueue events_power_efficient: flags=0x80 [ 7030.443499] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7030.451744] pending: neigh_periodic_work, do_cache_clean [ 7030.460471] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7030.463327] pending: neigh_periodic_work, check_lifetime [ 7030.465664] workqueue mm_percpu_wq: flags=0x8 [ 7030.468369] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7030.475880] pending: vmstat_update [ 7030.479167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7030.486968] pending: vmstat_update [ 7030.490949] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7030.500264] pending: vmstat_update [ 7030.502682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7030.505640] pending: vmstat_update [ 7030.507243] workqueue writeback: flags=0x4a [ 7030.508521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7030.514907] pending: wb_workfn, wb_workfn [ 7030.517395] workqueue xprtiod: flags=0xa [ 7030.521203] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7030.535160] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7030.555247] workqueue ipv6_addrconf: flags=0x40008 [ 7030.558045] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7030.561209] pending: addrconf_verify_work [ 7030.563221] workqueue knbd0-recv: flags=0x1a [ 7030.565041] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7030.570657] in-flight: 647:recv_work [ 7030.573390] workqueue ptlrpc_pinger: flags=0x2 [ 7030.575029] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7030.579981] pending: ptlrpc_pinger_main [ptlrpc] [ 7030.585591] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7020s workers=2 idle: 652 [ 7040.056176] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7040.060228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7040.096227] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7040.110516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7040.118628] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7040.121463] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7040.141399] RSP: 0018:ffffc38542073af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7040.150524] RAX: 0000000000007e1b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7040.161049] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7040.169276] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7040.184000] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7040.187900] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7040.199900] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7040.212121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7040.221564] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7040.226025] Call Trace: [ 7040.229272] [ 7040.232577] ? show_regs.cold.9+0x22/0x2f [ 7040.238373] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7040.244032] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7040.251492] ? watchdog+0x40/0x40 [ 7040.258138] ? hrtimer_interrupt+0x128/0x2c0 [ 7040.260232] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7040.262277] ? apic_timer_interrupt+0xf/0x20 [ 7040.264908] [ 7040.265926] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7040.270537] do_raw_spin_lock+0x177/0x190 [ 7040.273260] _raw_spin_lock+0x21/0x30 [ 7040.274622] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7040.278270] ? do_raw_spin_unlock+0x75/0x190 [ 7040.282356] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7040.285097] ? do_wait_intr+0xf0/0xf0 [ 7040.286716] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7040.289434] mdc_close+0x53b/0xd70 [mdc] [ 7040.291119] lmv_close+0x18d/0x490 [lmv] [ 7040.292500] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7040.298032] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7040.304076] ll_md_close+0x274/0x800 [lustre] [ 7040.306745] ll_file_release+0x128/0xc50 [lustre] [ 7040.308531] __fput+0x10d/0x440 [ 7040.311605] ____fput+0x12/0x20 [ 7040.312973] task_work_run+0xc6/0x110 [ 7040.314633] exit_to_usermode_loop+0x1dd/0x1f0 [ 7040.316306] do_syscall_64+0x3d6/0x440 [ 7040.324347] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7040.326470] RIP: 0033:0x7f1a63983cd8 [ 7040.329622] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7040.353020] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7040.359366] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7040.364037] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7040.368754] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7040.374466] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7040.377453] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7052.058037] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7052.061204] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7052.064643] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7052.082839] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7052.082856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7052.082859] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7052.082873] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7052.082877] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7052.082884] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7052.082886] RDX: 0000000000006e8a RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7052.082888] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7052.082890] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7052.082893] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7052.082896] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7052.082899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7052.082901] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7052.082910] Call Trace: [ 7052.082915] [ 7052.082920] ? show_regs.cold.9+0x22/0x2f [ 7052.082929] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7052.082936] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7052.082940] ? watchdog+0x40/0x40 [ 7052.082944] ? hrtimer_interrupt+0x128/0x2c0 [ 7052.082948] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7052.082956] ? apic_timer_interrupt+0xf/0x20 [ 7052.082960] [ 7052.082963] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7052.082968] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7052.082972] do_raw_spin_lock+0x177/0x190 [ 7052.082977] _raw_spin_lock+0x21/0x30 [ 7052.082984] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7052.083239] obd_mod_free+0x46/0x140 [ptlrpc] [ 7052.083738] mdc_commit_open+0x91/0xb0 [mdc] [ 7052.083775] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7052.083966] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7052.088128] Modules linked in: lustre(O) [ 7052.093664] after_reply+0xe2c/0x1780 [ptlrpc] [ 7052.097133] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [ 7052.100064] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7052.108648] lquota(O) [ 7052.112784] ? __next_timer_interrupt+0x160/0x160 [ 7052.112806] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7052.116539] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7052.116609] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7052.116616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7052.116619] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7052.116632] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7052.116636] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7052.116641] RAX: 0000000000000205 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7052.116643] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7052.116646] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7052.116648] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7052.116651] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7052.116654] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7052.116658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7052.116660] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7052.116669] Call Trace: [ 7052.124862] ? do_wait_intr+0xf0/0xf0 [ 7052.124884] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7052.125190] kthread+0x1d1/0x200 [ 7052.125197] ? set_kthread_struct+0x70/0x70 [ 7052.125201] ret_from_fork+0x1f/0x30 [ 7052.357128] [ 7052.358020] ? show_regs.cold.9+0x22/0x2f [ 7052.359623] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7052.361640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7052.364029] ? watchdog+0x40/0x40 [ 7052.365796] ? hrtimer_interrupt+0x128/0x2c0 [ 7052.367413] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7052.369415] ? apic_timer_interrupt+0xf/0x20 [ 7052.372084] [ 7052.373500] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7052.375369] do_raw_spin_lock+0x177/0x190 [ 7052.377358] _raw_spin_lock+0x21/0x30 [ 7052.378985] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7052.381615] ? do_raw_spin_unlock+0x75/0x190 [ 7052.383660] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7052.385843] ? do_wait_intr+0xf0/0xf0 [ 7052.387101] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7052.389135] mdc_close+0x53b/0xd70 [mdc] [ 7052.390482] lmv_close+0x18d/0x490 [lmv] [ 7052.395162] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7052.399685] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7052.401423] ll_md_close+0x274/0x800 [lustre] [ 7052.407922] ll_file_release+0x128/0xc50 [lustre] [ 7052.412212] __fput+0x10d/0x440 [ 7052.415089] ____fput+0x12/0x20 [ 7052.417966] task_work_run+0xc6/0x110 [ 7052.420385] exit_to_usermode_loop+0x1dd/0x1f0 [ 7052.423029] do_syscall_64+0x3d6/0x440 [ 7052.424303] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7052.427160] RIP: 0033:0x7f1a63983cd8 [ 7052.429792] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7052.439302] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7052.443446] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7052.447912] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7052.452441] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7052.455299] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7052.458863] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7056.053192] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7056.055906] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7056.097277] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7056.102750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7056.106569] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7056.109680] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7056.118292] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7056.122061] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7056.125121] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7056.130700] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7056.133267] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7056.140974] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7056.144739] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7056.150734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7056.154747] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7056.160571] Call Trace: [ 7056.163013] [ 7056.165046] ? show_regs.cold.9+0x22/0x2f [ 7056.166859] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7056.168785] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7056.170028] ? watchdog+0x40/0x40 [ 7056.171937] ? hrtimer_interrupt+0x128/0x2c0 [ 7056.173339] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7056.175594] ? apic_timer_interrupt+0xf/0x20 [ 7056.177618] [ 7056.181927] ? native_safe_halt+0x2e/0x50 [ 7056.183775] kvm_wait+0xcd/0xe0 [ 7056.184637] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7056.186896] do_raw_spin_lock+0x177/0x190 [ 7056.188941] _raw_spin_lock+0x21/0x30 [ 7056.190551] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7056.192416] ? do_raw_spin_unlock+0x75/0x190 [ 7056.194429] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7056.196498] ? do_wait_intr+0xf0/0xf0 [ 7056.197600] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7056.199131] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7056.200678] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7056.203240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7056.206577] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7056.209558] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7056.211309] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7056.214534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7056.217156] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7056.220498] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7056.222435] lmv_intent_open+0x338/0x1150 [lmv] [ 7056.224667] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7056.227193] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7056.229455] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7056.231083] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7056.233552] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7056.237276] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7056.239614] ? do_raw_spin_unlock+0x75/0x190 [ 7056.242944] ? _raw_spin_unlock+0x12/0x30 [ 7056.245905] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7056.249651] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7056.256308] ll_atomic_open+0x36a/0xd70 [lustre] [ 7056.261178] lookup_open+0x356/0xb40 [ 7056.264529] do_last+0x313/0xfc0 [ 7056.268424] ? nd_jump_root+0xe5/0x160 [ 7056.272498] ? path_init+0x437/0x520 [ 7056.274906] path_openat+0xf7/0x500 [ 7056.279398] ? do_raw_spin_unlock+0x75/0x190 [ 7056.283436] do_filp_open+0x99/0x140 [ 7056.290273] ? getname_flags+0x6e/0x330 [ 7056.292310] ? __check_object_size+0xff/0x256 [ 7056.295677] ? do_raw_spin_unlock+0x75/0x190 [ 7056.301368] ? _raw_spin_unlock+0x12/0x30 [ 7056.306256] do_sys_openat2+0x2b4/0x410 [ 7056.308225] do_sys_open+0x73/0xa0 [ 7056.310264] __x64_sys_openat+0x24/0x30 [ 7056.312413] do_syscall_64+0xc1/0x440 [ 7056.314471] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7056.319915] RIP: 0033:0x7f1a63983332 [ 7056.321934] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7056.334314] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7056.339628] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7056.343448] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7056.348931] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7056.353816] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7056.356809] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7060.961047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5037s! [ 7060.964678] Showing busy workqueues and worker pools: [ 7060.967029] workqueue events: flags=0x0 [ 7060.968808] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7060.971978] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7060.978022] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7060.988081] pending: sec_gc_main [ptlrpc] [ 7060.993560] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7060.996177] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7061.003166] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7061.006286] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7061.011982] workqueue events_long: flags=0x0 [ 7061.014265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7061.021902] pending: lu_objects_flush [obdclass] [ 7061.024604] workqueue events_unbound: flags=0x2 [ 7061.027028] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7061.032523] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7061.037970] workqueue events_power_efficient: flags=0x80 [ 7061.040180] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7061.045181] pending: neigh_periodic_work, do_cache_clean [ 7061.048964] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7061.052728] pending: neigh_periodic_work, check_lifetime [ 7061.057023] workqueue mm_percpu_wq: flags=0x8 [ 7061.062070] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7061.067656] pending: vmstat_update [ 7061.069688] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7061.075550] pending: vmstat_update [ 7061.076832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7061.080988] pending: vmstat_update [ 7061.083167] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7061.087053] pending: vmstat_update [ 7061.090839] workqueue writeback: flags=0x4a [ 7061.093295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7061.097297] pending: wb_workfn, wb_workfn [ 7061.101889] workqueue xprtiod: flags=0xa [ 7061.103521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7061.109295] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7061.113060] workqueue ipv6_addrconf: flags=0x40008 [ 7061.120925] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7061.123699] pending: addrconf_verify_work [ 7061.125598] workqueue knbd0-recv: flags=0x1a [ 7061.129603] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7061.134588] in-flight: 647:recv_work [ 7061.136703] workqueue ptlrpc_pinger: flags=0x2 [ 7061.141552] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7061.145430] pending: ptlrpc_pinger_main [ptlrpc] [ 7061.147953] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7051s workers=2 idle: 652 [ 7068.057435] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7068.063173] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7068.086107] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7068.090210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7068.093254] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7068.096085] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7068.102487] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7068.105418] RAX: 0000000000007f8a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7068.107876] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7068.110672] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7068.113389] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7068.116005] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7068.118721] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7068.121785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7068.123918] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7068.126731] Call Trace: [ 7068.127686] [ 7068.128579] ? show_regs.cold.9+0x22/0x2f [ 7068.130078] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7068.131934] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7068.133557] ? watchdog+0x40/0x40 [ 7068.134790] ? hrtimer_interrupt+0x128/0x2c0 [ 7068.136403] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7068.138121] ? apic_timer_interrupt+0xf/0x20 [ 7068.139660] [ 7068.140484] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7068.142495] do_raw_spin_lock+0x177/0x190 [ 7068.143908] _raw_spin_lock+0x21/0x30 [ 7068.145249] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7068.147117] ? do_raw_spin_unlock+0x75/0x190 [ 7068.148431] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7068.150358] ? do_wait_intr+0xf0/0xf0 [ 7068.151723] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7068.155469] mdc_close+0x53b/0xd70 [mdc] [ 7068.157025] lmv_close+0x18d/0x490 [lmv] [ 7068.158494] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7068.160878] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7068.162708] ll_md_close+0x274/0x800 [lustre] [ 7068.164562] ll_file_release+0x128/0xc50 [lustre] [ 7068.166347] __fput+0x10d/0x440 [ 7068.167745] ____fput+0x12/0x20 [ 7068.169143] task_work_run+0xc6/0x110 [ 7068.170493] exit_to_usermode_loop+0x1dd/0x1f0 [ 7068.172076] do_syscall_64+0x3d6/0x440 [ 7068.173485] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7068.175233] RIP: 0033:0x7f1a63983cd8 [ 7068.176585] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7068.183435] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7068.186364] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7068.189155] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7068.191930] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7068.194770] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7068.197515] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7080.057031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7080.058096] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7080.061025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel [ 7080.064892] Modules linked in: lustre(O) osp(O) [ 7080.081344] ata_generic ghash_clmulni_intel ata_piix serio_raw [ 7080.083440] ofd(O) lod(O) mdt(O) [ 7080.085762] libata dm_mirror [ 7080.087341] mdd(O) [ 7080.087343] dm_region_hash dm_log [ 7080.088598] mgs(O) [ 7080.089892] dm_mod [ 7080.091762] lquota(O) [ 7080.092711] sha512_ssse3 sha512_generic [ 7080.092721] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7080.092728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7080.092730] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7080.092744] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7080.092748] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7080.092752] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7080.092754] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7080.092756] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7080.092758] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7080.092761] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7080.092764] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7080.092767] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7080.092769] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7080.092778] Call Trace: [ 7080.092785] [ 7080.092791] ? show_regs.cold.9+0x22/0x2f [ 7080.092801] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7080.092808] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7080.092812] ? watchdog+0x40/0x40 [ 7080.092816] ? hrtimer_interrupt+0x128/0x2c0 [ 7080.092820] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7080.092827] ? apic_timer_interrupt+0xf/0x20 [ 7080.092831] [ 7080.092834] ? native_safe_halt+0x2e/0x50 [ 7080.092838] kvm_wait+0xcd/0xe0 [ 7080.092845] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7080.092856] do_raw_spin_lock+0x177/0x190 [ 7080.092861] _raw_spin_lock+0x21/0x30 [ 7080.092865] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7080.094339] lfsck(O) [ 7080.095374] ? do_raw_spin_unlock+0x75/0x190 [ 7080.098276] obdecho(O) [ 7080.103477] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7080.116724] mgc(O) [ 7080.119274] ? do_wait_intr+0xf0/0xf0 [ 7080.119304] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7080.119960] mdc_close+0x53b/0xd70 [mdc] [ 7080.120009] lmv_close+0x18d/0x490 [lmv] [ 7080.120063] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7080.120202] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7080.136196] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7080.136275] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7080.136283] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7080.136286] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7080.136303] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7080.136308] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7080.136312] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7080.136315] RDX: 0000000000004a5b RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7080.136317] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7080.136319] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7080.136321] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7080.136324] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7080.136327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7080.136329] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7080.136339] Call Trace: [ 7080.136345] [ 7080.136352] ? show_regs.cold.9+0x22/0x2f [ 7080.136363] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7080.136370] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7080.136375] ? watchdog+0x40/0x40 [ 7080.136380] ? hrtimer_interrupt+0x128/0x2c0 [ 7080.136383] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7080.136394] ? apic_timer_interrupt+0xf/0x20 [ 7080.136399] [ 7080.136402] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7080.136407] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7080.136411] do_raw_spin_lock+0x177/0x190 [ 7080.136416] _raw_spin_lock+0x21/0x30 [ 7080.136422] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7080.136798] obd_mod_free+0x46/0x140 [ptlrpc] [ 7080.137073] mdc_commit_open+0x91/0xb0 [mdc] [ 7080.137124] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7080.137377] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7080.137696] after_reply+0xe2c/0x1780 [ptlrpc] [ 7080.138107] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7080.138361] ? __next_timer_interrupt+0x160/0x160 [ 7080.138457] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7080.138927] ? do_wait_intr+0xf0/0xf0 [ 7080.138936] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7080.139141] kthread+0x1d1/0x200 [ 7080.139147] ? set_kthread_struct+0x70/0x70 [ 7080.139151] ret_from_fork+0x1f/0x30 [ 7080.389236] ll_md_close+0x274/0x800 [lustre] [ 7080.391494] ll_file_release+0x128/0xc50 [lustre] [ 7080.393957] __fput+0x10d/0x440 [ 7080.395428] ____fput+0x12/0x20 [ 7080.396806] task_work_run+0xc6/0x110 [ 7080.398923] exit_to_usermode_loop+0x1dd/0x1f0 [ 7080.401071] do_syscall_64+0x3d6/0x440 [ 7080.404131] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7080.406987] RIP: 0033:0x7f1a63983cd8 [ 7080.408966] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7080.417803] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7080.429156] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7080.440601] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7080.446043] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7080.451333] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7080.456863] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7088.053026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 7088.060615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7088.100152] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7088.104773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7088.108826] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7088.115605] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7088.127829] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7088.131945] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7088.140759] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7088.148483] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7088.153856] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7088.157613] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7088.161669] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7088.167558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7088.169564] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7088.172273] Call Trace: [ 7088.173103] [ 7088.173810] ? show_regs.cold.9+0x22/0x2f [ 7088.175251] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7088.176685] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7088.178652] ? watchdog+0x40/0x40 [ 7088.182635] ? hrtimer_interrupt+0x128/0x2c0 [ 7088.185149] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7088.188835] ? apic_timer_interrupt+0xf/0x20 [ 7088.191608] [ 7088.192263] ? native_safe_halt+0x2e/0x50 [ 7088.194452] kvm_wait+0xcd/0xe0 [ 7088.196388] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7088.197779] do_raw_spin_lock+0x177/0x190 [ 7088.198977] _raw_spin_lock+0x21/0x30 [ 7088.200291] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7088.203240] ? do_raw_spin_unlock+0x75/0x190 [ 7088.204999] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7088.207311] ? do_wait_intr+0xf0/0xf0 [ 7088.208876] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7088.211369] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7088.213864] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7088.216489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7088.218768] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7088.220989] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7088.222643] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7088.224688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7088.227131] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7088.231072] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7088.233531] lmv_intent_open+0x338/0x1150 [lmv] [ 7088.235511] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7088.237761] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7088.239486] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7088.240905] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7088.242863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7088.245150] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7088.247337] ? do_raw_spin_unlock+0x75/0x190 [ 7088.249125] ? _raw_spin_unlock+0x12/0x30 [ 7088.250887] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7088.253042] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7088.255416] ll_atomic_open+0x36a/0xd70 [lustre] [ 7088.257465] lookup_open+0x356/0xb40 [ 7088.258959] do_last+0x313/0xfc0 [ 7088.260292] ? nd_jump_root+0xe5/0x160 [ 7088.261864] ? path_init+0x437/0x520 [ 7088.263510] path_openat+0xf7/0x500 [ 7088.264961] ? do_raw_spin_unlock+0x75/0x190 [ 7088.266760] do_filp_open+0x99/0x140 [ 7088.268375] ? getname_flags+0x6e/0x330 [ 7088.269884] ? __check_object_size+0xff/0x256 [ 7088.271640] ? do_raw_spin_unlock+0x75/0x190 [ 7088.273363] ? _raw_spin_unlock+0x12/0x30 [ 7088.274858] do_sys_openat2+0x2b4/0x410 [ 7088.276478] do_sys_open+0x73/0xa0 [ 7088.277882] __x64_sys_openat+0x24/0x30 [ 7088.279564] do_syscall_64+0xc1/0x440 [ 7088.281042] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7088.283334] RIP: 0033:0x7f1a63983332 [ 7088.285028] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7088.293848] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7088.297340] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7088.300495] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7088.303649] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7088.306811] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7088.310111] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7091.169046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5068s! [ 7091.171540] Showing busy workqueues and worker pools: [ 7091.173520] workqueue events: flags=0x0 [ 7091.175837] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7091.187088] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7091.205234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7091.215335] pending: sec_gc_main [ptlrpc] [ 7091.222926] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7091.232048] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7091.240793] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7091.252453] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7091.265266] workqueue events_long: flags=0x0 [ 7091.266839] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7091.273976] pending: lu_objects_flush [obdclass] [ 7091.283465] workqueue events_unbound: flags=0x2 [ 7091.290502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7091.294459] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7091.307939] workqueue events_power_efficient: flags=0x80 [ 7091.312583] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7091.319342] pending: neigh_periodic_work, do_cache_clean [ 7091.329360] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7091.332001] pending: neigh_periodic_work, check_lifetime [ 7091.335647] workqueue mm_percpu_wq: flags=0x8 [ 7091.338655] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7091.346678] pending: vmstat_update [ 7091.357829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7091.373816] pending: vmstat_update [ 7091.376638] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7091.383787] pending: vmstat_update [ 7091.385249] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7091.389480] pending: vmstat_update [ 7091.390948] workqueue writeback: flags=0x4a [ 7091.399141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7091.406794] pending: wb_workfn, wb_workfn [ 7091.410595] workqueue xprtiod: flags=0xa [ 7091.411889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7091.414162] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7091.417384] workqueue ipv6_addrconf: flags=0x40008 [ 7091.419288] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7091.421950] pending: addrconf_verify_work [ 7091.423691] workqueue knbd0-recv: flags=0x1a [ 7091.425376] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7091.427884] in-flight: 647:recv_work [ 7091.431091] workqueue ptlrpc_pinger: flags=0x2 [ 7091.433380] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7091.436236] pending: ptlrpc_pinger_main [ptlrpc] [ 7091.440086] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7081s workers=2 idle: 652 [ 7096.056106] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7096.062881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7096.088606] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7096.095737] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7096.100600] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7096.104438] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7096.116706] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7096.123344] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7096.127471] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7096.131481] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7096.135245] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7096.140950] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7096.145828] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7096.151295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7096.154062] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7096.157737] Call Trace: [ 7096.159315] [ 7096.160166] ? show_regs.cold.9+0x22/0x2f [ 7096.161870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7096.164247] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7096.168731] ? watchdog+0x40/0x40 [ 7096.170435] ? hrtimer_interrupt+0x128/0x2c0 [ 7096.173917] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7096.176836] ? apic_timer_interrupt+0xf/0x20 [ 7096.179374] [ 7096.181383] ? native_safe_halt+0x2e/0x50 [ 7096.183334] kvm_wait+0xcd/0xe0 [ 7096.184731] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7096.189597] do_raw_spin_lock+0x177/0x190 [ 7096.192520] _raw_spin_lock+0x21/0x30 [ 7096.194519] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7096.198418] ? do_raw_spin_unlock+0x75/0x190 [ 7096.203521] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7096.208442] ? do_wait_intr+0xf0/0xf0 [ 7096.213617] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7096.219697] mdc_close+0x53b/0xd70 [mdc] [ 7096.225980] lmv_close+0x18d/0x490 [lmv] [ 7096.227573] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7096.229186] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7096.235007] ll_md_close+0x274/0x800 [lustre] [ 7096.237725] ll_file_release+0x128/0xc50 [lustre] [ 7096.239362] __fput+0x10d/0x440 [ 7096.240392] ____fput+0x12/0x20 [ 7096.241345] task_work_run+0xc6/0x110 [ 7096.243683] exit_to_usermode_loop+0x1dd/0x1f0 [ 7096.245677] do_syscall_64+0x3d6/0x440 [ 7096.248806] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7096.250690] RIP: 0033:0x7f1a63983cd8 [ 7096.253869] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7096.264063] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7096.268253] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7096.272203] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7096.275627] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7096.280310] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7096.283460] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7108.057453] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7108.058008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7108.067454] Modules linked in: [ 7108.073507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7108.073583] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7108.073590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7108.073593] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7108.073606] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7108.073610] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7108.073614] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7108.073616] RDX: 0000000000004384 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7108.073619] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7108.073621] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7108.073623] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7108.073626] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7108.073629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7108.073631] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7108.073639] Call Trace: [ 7108.073644] [ 7108.073648] ? show_regs.cold.9+0x22/0x2f [ 7108.073656] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7108.073661] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7108.073665] ? watchdog+0x40/0x40 [ 7108.073669] ? hrtimer_interrupt+0x128/0x2c0 [ 7108.073673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7108.073681] ? apic_timer_interrupt+0xf/0x20 [ 7108.073686] [ 7108.073689] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7108.073694] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7108.073699] do_raw_spin_lock+0x177/0x190 [ 7108.073703] _raw_spin_lock+0x21/0x30 [ 7108.073708] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7108.074040] obd_mod_free+0x46/0x140 [ptlrpc] [ 7108.079569] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7108.081404] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7108.081412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7108.081416] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7108.081428] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7108.081432] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7108.081437] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7108.081440] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7108.081442] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7108.081445] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7108.081447] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7108.081450] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7108.081453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7108.081456] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7108.081465] Call Trace: [ 7108.081470] [ 7108.081476] ? show_regs.cold.9+0x22/0x2f [ 7108.081486] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7108.081492] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7108.081496] ? watchdog+0x40/0x40 [ 7108.081500] ? hrtimer_interrupt+0x128/0x2c0 [ 7108.081505] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7108.081512] ? apic_timer_interrupt+0xf/0x20 [ 7108.081517] [ 7108.081520] ? native_safe_halt+0x2e/0x50 [ 7108.081525] kvm_wait+0xcd/0xe0 [ 7108.081530] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7108.081538] do_raw_spin_lock+0x177/0x190 [ 7108.081543] _raw_spin_lock+0x21/0x30 [ 7108.081547] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7108.085946] ? do_raw_spin_unlock+0x75/0x190 [ 7108.085956] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7108.086611] ? do_wait_intr+0xf0/0xf0 [ 7108.086620] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7108.122962] mdc_commit_open+0x91/0xb0 [mdc] [ 7108.123025] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7108.123520] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7108.123930] after_reply+0xe2c/0x1780 [ptlrpc] [ 7108.124134] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7108.124328] ? __next_timer_interrupt+0x160/0x160 [ 7108.124339] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7108.124553] ? do_wait_intr+0xf0/0xf0 [ 7108.124570] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7108.124896] kthread+0x1d1/0x200 [ 7108.124903] ? set_kthread_struct+0x70/0x70 [ 7108.124907] ret_from_fork+0x1f/0x30 [ 7108.484597] mdc_close+0x53b/0xd70 [mdc] [ 7108.488951] lmv_close+0x18d/0x490 [lmv] [ 7108.490455] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7108.498160] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7108.502678] ll_md_close+0x274/0x800 [lustre] [ 7108.505447] ll_file_release+0x128/0xc50 [lustre] [ 7108.511652] __fput+0x10d/0x440 [ 7108.516369] ____fput+0x12/0x20 [ 7108.522594] task_work_run+0xc6/0x110 [ 7108.526460] exit_to_usermode_loop+0x1dd/0x1f0 [ 7108.530606] do_syscall_64+0x3d6/0x440 [ 7108.532536] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7108.541788] RIP: 0033:0x7f1a63983cd8 [ 7108.543122] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7108.572030] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7108.578403] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7108.583066] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7108.588572] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7108.597365] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7108.604268] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7116.053009] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 7116.056318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7116.106582] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7116.113424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7116.123874] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7116.129503] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7116.149542] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7116.163451] RAX: 0000000000006f6a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7116.169569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7116.178072] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7116.183234] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7116.196664] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7116.201027] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7116.207366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7116.217670] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7116.229631] Call Trace: [ 7116.234667] [ 7116.247149] ? show_regs.cold.9+0x22/0x2f [ 7116.261910] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7116.263660] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7116.271471] ? watchdog+0x40/0x40 [ 7116.274682] ? hrtimer_interrupt+0x128/0x2c0 [ 7116.283847] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7116.294245] ? apic_timer_interrupt+0xf/0x20 [ 7116.299229] [ 7116.300993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7116.304093] do_raw_spin_lock+0x177/0x190 [ 7116.310645] _raw_spin_lock+0x21/0x30 [ 7116.317965] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7116.330181] ? do_raw_spin_unlock+0x75/0x190 [ 7116.338456] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7116.342525] ? do_wait_intr+0xf0/0xf0 [ 7116.345635] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7116.347716] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7116.363354] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7116.370231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7116.381861] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7116.395065] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7116.396876] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7116.400817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7116.413372] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7116.426107] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7116.429148] lmv_intent_open+0x338/0x1150 [lmv] [ 7116.436393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7116.447445] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7116.462172] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7116.466654] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7116.474252] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7116.479704] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7116.481437] ? do_raw_spin_unlock+0x75/0x190 [ 7116.491592] ? _raw_spin_unlock+0x12/0x30 [ 7116.499521] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7116.511985] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7116.519060] ll_atomic_open+0x36a/0xd70 [lustre] [ 7116.531098] lookup_open+0x356/0xb40 [ 7116.534100] do_last+0x313/0xfc0 [ 7116.540198] ? nd_jump_root+0xe5/0x160 [ 7116.544350] ? path_init+0x437/0x520 [ 7116.547343] path_openat+0xf7/0x500 [ 7116.551364] ? do_raw_spin_unlock+0x75/0x190 [ 7116.562402] do_filp_open+0x99/0x140 [ 7116.565472] ? getname_flags+0x6e/0x330 [ 7116.571392] ? __check_object_size+0xff/0x256 [ 7116.577209] ? do_raw_spin_unlock+0x75/0x190 [ 7116.586280] ? _raw_spin_unlock+0x12/0x30 [ 7116.591347] do_sys_openat2+0x2b4/0x410 [ 7116.596977] do_sys_open+0x73/0xa0 [ 7116.604353] __x64_sys_openat+0x24/0x30 [ 7116.610360] do_syscall_64+0xc1/0x440 [ 7116.630913] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7116.632778] RIP: 0033:0x7f1a63983332 [ 7116.638175] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7116.673211] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7116.684395] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7116.693970] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7116.702460] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7116.710255] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7116.718144] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7121.893592] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5098s! [ 7121.902362] Showing busy workqueues and worker pools: [ 7121.908302] workqueue events: flags=0x0 [ 7121.911305] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7121.914806] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7121.921063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7121.927286] pending: sec_gc_main [ptlrpc] [ 7121.935326] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7121.940007] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7121.945143] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7121.949320] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7121.958140] workqueue events_long: flags=0x0 [ 7121.965617] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7121.974258] pending: lu_objects_flush [obdclass] [ 7121.985413] workqueue events_unbound: flags=0x2 [ 7121.990282] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7121.995431] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7122.001427] workqueue events_power_efficient: flags=0x80 [ 7122.008481] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7122.019830] pending: neigh_periodic_work, do_cache_clean [ 7122.029065] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7122.035671] pending: neigh_periodic_work, check_lifetime [ 7122.038415] workqueue mm_percpu_wq: flags=0x8 [ 7122.041835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7122.056190] pending: vmstat_update [ 7122.057631] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7122.071708] pending: vmstat_update [ 7122.083777] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7122.086325] pending: vmstat_update [ 7122.097010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7122.106508] pending: vmstat_update [ 7122.108079] workqueue writeback: flags=0x4a [ 7122.112246] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7122.125674] pending: wb_workfn, wb_workfn [ 7122.133472] workqueue xprtiod: flags=0xa [ 7122.137259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7122.147313] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7122.163712] workqueue ipv6_addrconf: flags=0x40008 [ 7122.166117] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7122.169047] pending: addrconf_verify_work [ 7122.172404] workqueue knbd0-recv: flags=0x1a [ 7122.182940] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7122.185320] in-flight: 647:recv_work [ 7122.188567] workqueue ptlrpc_pinger: flags=0x2 [ 7122.192831] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7122.195825] pending: ptlrpc_pinger_main [ptlrpc] [ 7122.198808] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7112s workers=2 idle: 652 [ 7122.526304] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 7122.531992] rcu: 3-...!: (3171287 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 7122.543957] (detected by 1, t=5100120 jiffies, g=269633, q=1893) [ 7122.551033] Sending NMI from CPU 1 to CPUs 3: [ 7122.558170] NMI backtrace for cpu 3 [ 7122.558176] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7122.558177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7122.558179] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7122.558180] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7122.558182] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 7122.558183] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7122.558184] RDX: 0000000000000649 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7122.558185] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7122.558186] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7122.558187] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7122.558188] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7122.558189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7122.558189] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7122.558190] Call Trace: [ 7122.558191] [ 7122.558191] ? show_regs.cold.9+0x22/0x2f [ 7122.558192] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7122.558192] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7122.558193] ? nmi_handle+0xbd/0x210 [ 7122.558194] ? default_do_nmi+0x4b/0x1b0 [ 7122.558194] ? do_nmi+0x280/0x380 [ 7122.558195] ? end_repeat_nmi+0x16/0x1f [ 7122.558195] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7122.558196] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7122.558197] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7122.558197] [ 7122.558198] do_raw_spin_lock+0x177/0x190 [ 7122.558198] _raw_spin_lock+0x21/0x30 [ 7122.558199] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7122.558199] obd_mod_free+0x46/0x140 [ptlrpc] [ 7122.558200] mdc_commit_open+0x91/0xb0 [mdc] [ 7122.558201] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7122.558201] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7122.558202] after_reply+0xe2c/0x1780 [ptlrpc] [ 7122.558202] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7122.558203] ? __next_timer_interrupt+0x160/0x160 [ 7122.558203] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7122.558204] ? do_wait_intr+0xf0/0xf0 [ 7122.558205] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7122.558205] kthread+0x1d1/0x200 [ 7122.558206] ? set_kthread_struct+0x70/0x70 [ 7122.558206] ret_from_fork+0x1f/0x30 [ 7122.558821] rcu: rcu_sched kthread starved for 5099766 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 7122.756819] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7122.767619] rcu: RCU grace-period kthread stack dump: [ 7122.775416] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7122.783756] Call Trace: [ 7122.791262] __schedule+0x351/0xcb0 [ 7122.797432] schedule+0xc0/0x180 [ 7122.804391] schedule_timeout+0xb4/0x190 [ 7122.810727] ? do_raw_spin_unlock+0x75/0x190 [ 7122.821306] ? __next_timer_interrupt+0x160/0x160 [ 7122.822803] ? prepare_to_swait_event+0xf3/0x140 [ 7122.829780] rcu_gp_kthread+0x8dd/0x15b0 [ 7122.836583] ? force_qs_rnp+0x2c0/0x2c0 [ 7122.839732] kthread+0x1d1/0x200 [ 7122.850023] ? set_kthread_struct+0x70/0x70 [ 7122.852582] ret_from_fork+0x1f/0x30 [ 7122.855289] rcu: Stack dump where RCU GP kthread last ran: [ 7122.859533] NMI backtrace for cpu 1 [ 7122.862329] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7122.879369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7122.882660] Call Trace: [ 7122.884645] [ 7122.885605] ? dump_stack+0xbb/0x10e [ 7122.887863] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7122.893719] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7122.897595] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7122.904011] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7122.912050] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7122.917732] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 7122.933082] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 7122.937295] ? update_process_times+0x6d/0xc0 [ 7122.942669] ? tick_sched_handle.isra.18+0x27/0x90 [ 7122.949178] ? tick_sched_timer+0x53/0xb0 [ 7122.959148] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7122.960506] ? tick_sched_handle.isra.18+0x90/0x90 [ 7122.975241] ? hrtimer_interrupt+0x128/0x2c0 [ 7122.984457] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7122.995369] ? apic_timer_interrupt+0xf/0x20 [ 7122.996732] [ 7123.000632] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7123.003820] ? do_raw_spin_lock+0x177/0x190 [ 7123.011384] ? _raw_spin_lock+0x21/0x30 [ 7123.012634] ? ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7123.023773] ? do_raw_spin_unlock+0x75/0x190 [ 7123.030508] ? ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7123.038559] ? do_wait_intr+0xf0/0xf0 [ 7123.046007] ? ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7123.053730] ? mdc_close+0x53b/0xd70 [mdc] [ 7123.057243] ? lmv_close+0x18d/0x490 [lmv] [ 7123.060323] ? ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7123.065447] ? ll_md_real_close+0xa5/0x2a0 [lustre] [ 7123.071444] ? ll_md_close+0x274/0x800 [lustre] [ 7123.073559] ? ll_file_release+0x128/0xc50 [lustre] [ 7123.081354] ? __fput+0x10d/0x440 [ 7123.082638] ? ____fput+0x12/0x20 [ 7123.084234] ? task_work_run+0xc6/0x110 [ 7123.086109] ? exit_to_usermode_loop+0x1dd/0x1f0 [ 7123.090695] ? do_syscall_64+0x3d6/0x440 [ 7123.091950] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7136.057019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7136.063944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7136.108246] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7136.115050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7136.121165] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7136.124924] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7136.138895] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7136.143326] RAX: 0000000000007154 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7136.149149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7136.154365] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7136.159156] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7136.164968] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7136.170555] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7136.175585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7136.178819] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7136.183492] Call Trace: [ 7136.185374] [ 7136.186584] ? show_regs.cold.9+0x22/0x2f [ 7136.189862] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7136.192505] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7136.196472] ? watchdog+0x40/0x40 [ 7136.198745] ? hrtimer_interrupt+0x128/0x2c0 [ 7136.203778] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7136.207602] ? apic_timer_interrupt+0xf/0x20 [ 7136.215373] [ 7136.220462] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7136.226304] do_raw_spin_lock+0x177/0x190 [ 7136.229627] _raw_spin_lock+0x21/0x30 [ 7136.231463] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7136.236748] ? do_raw_spin_unlock+0x75/0x190 [ 7136.238482] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7136.241022] ? do_wait_intr+0xf0/0xf0 [ 7136.243462] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7136.246502] mdc_close+0x53b/0xd70 [mdc] [ 7136.249315] lmv_close+0x18d/0x490 [lmv] [ 7136.251861] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7136.254859] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7136.257861] ll_md_close+0x274/0x800 [lustre] [ 7136.264575] ll_file_release+0x128/0xc50 [lustre] [ 7136.268015] __fput+0x10d/0x440 [ 7136.270651] ____fput+0x12/0x20 [ 7136.271813] task_work_run+0xc6/0x110 [ 7136.274529] exit_to_usermode_loop+0x1dd/0x1f0 [ 7136.276489] do_syscall_64+0x3d6/0x440 [ 7136.277689] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7136.279366] RIP: 0033:0x7f1a63983cd8 [ 7136.280865] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7136.288527] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7136.291422] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7136.294075] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7136.296627] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7136.298994] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7136.301380] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7148.053923] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 7148.058018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 7148.058296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7148.058354] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7148.058360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7148.058363] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7148.058377] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7148.058380] RSP: 0018:ffffc38541c37500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7148.058385] RAX: 0000000000004d31 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7148.058387] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7148.058390] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7148.058392] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7148.058394] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7148.058397] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7148.058400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7148.058402] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7148.058411] Call Trace: [ 7148.058416] [ 7148.058421] ? show_regs.cold.9+0x22/0x2f [ 7148.058430] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7148.058436] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7148.058440] ? watchdog+0x40/0x40 [ 7148.058444] ? hrtimer_interrupt+0x128/0x2c0 [ 7148.058448] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7148.058455] ? apic_timer_interrupt+0xf/0x20 [ 7148.058459] [ 7148.058462] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7148.058468] do_raw_spin_lock+0x177/0x190 [ 7148.058472] _raw_spin_lock+0x21/0x30 [ 7148.058476] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7148.058758] ? do_raw_spin_unlock+0x75/0x190 [ 7148.058768] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 7148.058774] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7148.058833] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7148.058837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7148.058840] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7148.058847] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7148.058851] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7148.058855] RAX: 000000000000750c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7148.058857] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7148.058764] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7148.058861] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7148.058865] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7148.058867] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7148.058980] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7148.058984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7148.058986] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7148.058993] Call Trace: [ 7148.058996] [ 7148.058999] ? show_regs.cold.9+0x22/0x2f [ 7148.059006] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7148.059012] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7148.059109] ? watchdog+0x40/0x40 [ 7148.059115] ? hrtimer_interrupt+0x128/0x2c0 [ 7148.059120] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7148.059126] ? apic_timer_interrupt+0xf/0x20 [ 7148.059131] [ 7148.059134] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7148.059140] do_raw_spin_lock+0x177/0x190 [ 7148.059145] _raw_spin_lock+0x21/0x30 [ 7148.059149] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7148.059374] ? do_raw_spin_unlock+0x75/0x190 [ 7148.059380] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7148.059830] ? do_wait_intr+0xf0/0xf0 [ 7148.059838] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7148.060334] mdc_close+0x53b/0xd70 [mdc] [ 7148.060370] lmv_close+0x18d/0x490 [lmv] [ 7148.060403] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7148.060522] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7148.060620] ll_md_close+0x274/0x800 [lustre] [ 7148.060812] ll_file_release+0x128/0xc50 [lustre] [ 7148.060980] __fput+0x10d/0x440 [ 7148.060989] ____fput+0x12/0x20 [ 7148.060994] task_work_run+0xc6/0x110 [ 7148.061000] exit_to_usermode_loop+0x1dd/0x1f0 [ 7148.061008] do_syscall_64+0x3d6/0x440 [ 7148.061014] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7148.061146] RIP: 0033:0x7f1a63983cd8 [ 7148.061153] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7148.061157] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7148.061161] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7148.061163] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7148.061165] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7148.061167] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7148.061169] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7148.062912] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7148.062998] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7148.063004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7148.063007] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7148.063018] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7148.063023] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7148.063028] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7148.063030] RDX: 000000000000388d RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7148.063033] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7148.063035] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7148.063037] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7148.063041] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7148.063044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7148.063046] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7148.063053] Call Trace: [ 7148.063057] [ 7148.063061] ? show_regs.cold.9+0x22/0x2f [ 7148.063069] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7148.063076] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7148.063081] ? watchdog+0x40/0x40 [ 7148.063085] ? hrtimer_interrupt+0x128/0x2c0 [ 7148.063095] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7148.063104] ? apic_timer_interrupt+0xf/0x20 [ 7148.063109] [ 7148.063112] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7148.063116] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7148.063121] do_raw_spin_lock+0x177/0x190 [ 7148.063125] _raw_spin_lock+0x21/0x30 [ 7148.063130] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7148.063394] obd_mod_free+0x46/0x140 [ptlrpc] [ 7148.063612] mdc_commit_open+0x91/0xb0 [mdc] [ 7148.063645] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7148.063873] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7148.064069] after_reply+0xe2c/0x1780 [ptlrpc] [ 7148.064366] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7148.064564] ? __next_timer_interrupt+0x160/0x160 [ 7148.064578] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7148.064995] ? do_wait_intr+0xf0/0xf0 [ 7148.065004] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7148.065208] kthread+0x1d1/0x200 [ 7148.065214] ? set_kthread_struct+0x70/0x70 [ 7148.065217] ret_from_fork+0x1f/0x30 [ 7148.492306] ? do_wait_intr+0xf0/0xf0 [ 7148.494421] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7148.497085] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7148.498583] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7148.500951] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7148.502476] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7148.504195] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7148.505505] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7148.507096] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7148.508763] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7148.510729] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7148.512442] lmv_intent_open+0x338/0x1150 [lmv] [ 7148.513718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7148.515568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7148.517711] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7148.519400] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7148.521095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7148.523209] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7148.524686] ? do_raw_spin_unlock+0x75/0x190 [ 7148.527093] ? _raw_spin_unlock+0x12/0x30 [ 7148.528729] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7148.530230] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7148.531972] ll_atomic_open+0x36a/0xd70 [lustre] [ 7148.535129] lookup_open+0x356/0xb40 [ 7148.536423] do_last+0x313/0xfc0 [ 7148.538021] ? nd_jump_root+0xe5/0x160 [ 7148.540492] ? path_init+0x437/0x520 [ 7148.541616] path_openat+0xf7/0x500 [ 7148.542737] ? do_raw_spin_unlock+0x75/0x190 [ 7148.547559] do_filp_open+0x99/0x140 [ 7148.553011] ? getname_flags+0x6e/0x330 [ 7148.556335] ? __check_object_size+0xff/0x256 [ 7148.558975] ? do_raw_spin_unlock+0x75/0x190 [ 7148.561452] ? _raw_spin_unlock+0x12/0x30 [ 7148.563036] do_sys_openat2+0x2b4/0x410 [ 7148.565131] do_sys_open+0x73/0xa0 [ 7148.566465] __x64_sys_openat+0x24/0x30 [ 7148.568713] do_syscall_64+0xc1/0x440 [ 7148.571007] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7148.573182] RIP: 0033:0x7f1a63983332 [ 7148.574564] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7148.586996] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7148.593749] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7148.597925] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7148.602290] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7148.604693] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7148.607503] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7152.609050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5129s! [ 7152.614220] Showing busy workqueues and worker pools: [ 7152.623515] workqueue events: flags=0x0 [ 7152.633721] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7152.641909] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7152.647131] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7152.656243] pending: sec_gc_main [ptlrpc] [ 7152.661233] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7152.666482] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7152.673967] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7152.677972] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7152.683989] workqueue events_long: flags=0x0 [ 7152.686312] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7152.692271] pending: lu_objects_flush [obdclass] [ 7152.696088] workqueue events_unbound: flags=0x2 [ 7152.701156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7152.706096] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7152.719413] workqueue events_power_efficient: flags=0x80 [ 7152.724109] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7152.729457] pending: neigh_periodic_work, do_cache_clean [ 7152.735182] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7152.745261] pending: neigh_periodic_work, check_lifetime [ 7152.748405] workqueue mm_percpu_wq: flags=0x8 [ 7152.754113] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7152.758973] pending: vmstat_update [ 7152.767051] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7152.773045] pending: vmstat_update [ 7152.777419] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7152.784070] pending: vmstat_update [ 7152.785515] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7152.792422] pending: vmstat_update [ 7152.796185] workqueue writeback: flags=0x4a [ 7152.802143] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7152.814969] pending: wb_workfn, wb_workfn [ 7152.817611] workqueue xprtiod: flags=0xa [ 7152.821174] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7152.829235] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7152.842914] workqueue ipv6_addrconf: flags=0x40008 [ 7152.847180] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7152.855720] pending: addrconf_verify_work [ 7152.864304] workqueue knbd0-recv: flags=0x1a [ 7152.870321] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7152.875913] in-flight: 647:recv_work [ 7152.878058] workqueue ptlrpc_pinger: flags=0x2 [ 7152.880514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7152.885396] pending: ptlrpc_pinger_main [ptlrpc] [ 7152.888967] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7142s workers=2 idle: 652 [ 7164.057598] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 7164.068316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7164.116967] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7164.122881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7164.125997] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7164.128366] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7164.148720] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7164.154060] RAX: 000000000000490b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7164.159390] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7164.168212] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7164.173729] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7164.176659] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7164.178357] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7164.183638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7164.185008] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7164.191505] Call Trace: [ 7164.194547] [ 7164.195829] ? show_regs.cold.9+0x22/0x2f [ 7164.197679] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7164.199198] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7164.202820] ? watchdog+0x40/0x40 [ 7164.205619] ? hrtimer_interrupt+0x128/0x2c0 [ 7164.208419] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7164.214311] ? apic_timer_interrupt+0xf/0x20 [ 7164.217789] [ 7164.218738] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7164.224150] do_raw_spin_lock+0x177/0x190 [ 7164.226294] _raw_spin_lock+0x21/0x30 [ 7164.228613] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7164.231283] ? do_raw_spin_unlock+0x75/0x190 [ 7164.233555] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7164.237832] ? do_wait_intr+0xf0/0xf0 [ 7164.240005] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7164.243392] mdc_close+0x53b/0xd70 [mdc] [ 7164.244687] lmv_close+0x18d/0x490 [lmv] [ 7164.245666] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7164.249700] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7164.255398] ll_md_close+0x274/0x800 [lustre] [ 7164.258256] ll_file_release+0x128/0xc50 [lustre] [ 7164.261157] __fput+0x10d/0x440 [ 7164.263337] ____fput+0x12/0x20 [ 7164.265143] task_work_run+0xc6/0x110 [ 7164.266860] exit_to_usermode_loop+0x1dd/0x1f0 [ 7164.270841] do_syscall_64+0x3d6/0x440 [ 7164.275128] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7164.277877] RIP: 0033:0x7f1a63983cd8 [ 7164.278988] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7164.289559] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7164.292415] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7164.295969] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7164.299382] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7164.302955] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7164.306878] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7176.057645] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 7176.058021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 7176.068141] Modules linked in: lustre(O) [ 7176.071125] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7176.071300] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7176.071311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7176.071314] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7176.071346] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7176.071349] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7176.071354] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7176.071357] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7176.071359] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7176.071361] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7176.071364] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7176.071367] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7176.071635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7176.071638] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7176.071645] Call Trace: [ 7176.071664] [ 7176.071669] ? show_regs.cold.9+0x22/0x2f [ 7176.071678] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7176.071685] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7176.071689] ? watchdog+0x40/0x40 [ 7176.071693] ? hrtimer_interrupt+0x128/0x2c0 [ 7176.071697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7176.071703] ? apic_timer_interrupt+0xf/0x20 [ 7176.071707] [ 7176.071710] ? native_safe_halt+0x2e/0x50 [ 7176.071714] kvm_wait+0xcd/0xe0 [ 7176.071720] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7176.071731] do_raw_spin_lock+0x177/0x190 [ 7176.071736] _raw_spin_lock+0x21/0x30 [ 7176.071742] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7176.072075] obd_mod_free+0x46/0x140 [ptlrpc] [ 7176.072294] mdc_commit_open+0x91/0xb0 [mdc] [ 7176.072362] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7176.072736] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7176.072992] after_reply+0xe2c/0x1780 [ptlrpc] [ 7176.073198] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7176.073411] ? __next_timer_interrupt+0x160/0x160 [ 7176.073420] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7176.073711] ? do_wait_intr+0xf0/0xf0 [ 7176.073720] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7176.073937] kthread+0x1d1/0x200 [ 7176.073944] ? set_kthread_struct+0x70/0x70 [ 7176.073948] ret_from_fork+0x1f/0x30 [ 7176.257375] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7176.282031] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7176.288565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7176.293034] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7176.296417] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7176.306520] RSP: 0018:ffffc38542073af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7176.310873] RAX: 0000000000002622 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7176.315224] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7176.319296] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7176.324326] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7176.329627] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7176.333828] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7176.338703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7176.342779] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7176.347021] Call Trace: [ 7176.348889] [ 7176.349924] ? show_regs.cold.9+0x22/0x2f [ 7176.352442] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7176.356193] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7176.358602] ? watchdog+0x40/0x40 [ 7176.360912] ? hrtimer_interrupt+0x128/0x2c0 [ 7176.363261] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7176.365503] ? apic_timer_interrupt+0xf/0x20 [ 7176.367516] [ 7176.368610] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7176.371477] do_raw_spin_lock+0x177/0x190 [ 7176.373577] _raw_spin_lock+0x21/0x30 [ 7176.375632] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7176.378900] ? do_raw_spin_unlock+0x75/0x190 [ 7176.380633] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7176.384067] ? do_wait_intr+0xf0/0xf0 [ 7176.386280] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7176.389808] mdc_close+0x53b/0xd70 [mdc] [ 7176.392084] lmv_close+0x18d/0x490 [lmv] [ 7176.394173] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7176.397410] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7176.399897] ll_md_close+0x274/0x800 [lustre] [ 7176.402508] ll_file_release+0x128/0xc50 [lustre] [ 7176.405571] __fput+0x10d/0x440 [ 7176.407281] ____fput+0x12/0x20 [ 7176.409371] task_work_run+0xc6/0x110 [ 7176.411258] exit_to_usermode_loop+0x1dd/0x1f0 [ 7176.413716] do_syscall_64+0x3d6/0x440 [ 7176.415566] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7176.418020] RIP: 0033:0x7f1a63983cd8 [ 7176.419771] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7176.430916] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7176.439791] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7176.443454] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7176.452182] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7176.462952] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7176.470940] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7180.054062] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7180.063991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7180.091041] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7180.100104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7180.109789] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7180.113022] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7180.127781] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7180.132047] RAX: 0000000000004a1a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7180.138684] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7180.142827] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7180.148075] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7180.152403] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7180.161884] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7180.164655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7180.167001] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7180.172951] Call Trace: [ 7180.174805] [ 7180.175822] ? show_regs.cold.9+0x22/0x2f [ 7180.177008] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7180.180100] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7180.182684] ? watchdog+0x40/0x40 [ 7180.183920] ? hrtimer_interrupt+0x128/0x2c0 [ 7180.187683] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7180.190760] ? apic_timer_interrupt+0xf/0x20 [ 7180.192614] [ 7180.193169] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7180.196385] do_raw_spin_lock+0x177/0x190 [ 7180.198100] _raw_spin_lock+0x21/0x30 [ 7180.201880] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7180.206049] ? do_raw_spin_unlock+0x75/0x190 [ 7180.209971] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7180.214636] ? do_wait_intr+0xf0/0xf0 [ 7180.216028] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7180.227720] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7180.236090] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7180.242980] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7180.250571] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7180.253738] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7180.258524] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7180.260678] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7180.267040] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7180.269901] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7180.273050] lmv_intent_open+0x338/0x1150 [lmv] [ 7180.282071] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7180.289646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7180.300008] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7180.302109] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7180.305696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7180.310821] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7180.315892] ? do_raw_spin_unlock+0x75/0x190 [ 7180.320678] ? _raw_spin_unlock+0x12/0x30 [ 7180.326840] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7180.331866] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7180.335833] ll_atomic_open+0x36a/0xd70 [lustre] [ 7180.339344] lookup_open+0x356/0xb40 [ 7180.345989] do_last+0x313/0xfc0 [ 7180.347964] ? nd_jump_root+0xe5/0x160 [ 7180.349734] ? path_init+0x437/0x520 [ 7180.352548] path_openat+0xf7/0x500 [ 7180.354455] ? do_raw_spin_unlock+0x75/0x190 [ 7180.356411] do_filp_open+0x99/0x140 [ 7180.358385] ? getname_flags+0x6e/0x330 [ 7180.360232] ? __check_object_size+0xff/0x256 [ 7180.362249] ? do_raw_spin_unlock+0x75/0x190 [ 7180.364374] ? _raw_spin_unlock+0x12/0x30 [ 7180.366210] do_sys_openat2+0x2b4/0x410 [ 7180.368093] do_sys_open+0x73/0xa0 [ 7180.369855] __x64_sys_openat+0x24/0x30 [ 7180.371731] do_syscall_64+0xc1/0x440 [ 7180.373451] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7180.375470] RIP: 0033:0x7f1a63983332 [ 7180.377073] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7180.385170] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7180.388393] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7180.391706] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7180.394784] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7180.397665] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7180.400436] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7183.328082] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5160s! [ 7183.330845] Showing busy workqueues and worker pools: [ 7183.332483] workqueue events: flags=0x0 [ 7183.334508] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7183.337147] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7183.340651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7183.343625] pending: sec_gc_main [ptlrpc] [ 7183.345977] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7183.348145] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7183.351610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7183.354151] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7183.357550] workqueue events_long: flags=0x0 [ 7183.359108] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7183.360993] pending: lu_objects_flush [obdclass] [ 7183.362946] workqueue events_unbound: flags=0x2 [ 7183.364305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7183.366032] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7183.369500] workqueue events_power_efficient: flags=0x80 [ 7183.371188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7183.373515] pending: neigh_periodic_work, do_cache_clean [ 7183.375063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7183.376918] pending: neigh_periodic_work, check_lifetime [ 7183.383786] workqueue mm_percpu_wq: flags=0x8 [ 7183.385150] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7183.391161] pending: vmstat_update [ 7183.398816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7183.408499] pending: vmstat_update [ 7183.410096] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7183.413852] pending: vmstat_update [ 7183.415112] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7183.420200] pending: vmstat_update [ 7183.421932] workqueue writeback: flags=0x4a [ 7183.423430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7183.426545] pending: wb_workfn, wb_workfn [ 7183.428706] workqueue xprtiod: flags=0xa [ 7183.429936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7183.433798] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7183.438175] workqueue ipv6_addrconf: flags=0x40008 [ 7183.440887] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7183.443744] pending: addrconf_verify_work [ 7183.445678] workqueue knbd0-recv: flags=0x1a [ 7183.449722] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7183.452559] in-flight: 647:recv_work [ 7183.453812] workqueue ptlrpc_pinger: flags=0x2 [ 7183.458502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7183.467593] pending: ptlrpc_pinger_main [ptlrpc] [ 7183.470507] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7173s workers=2 idle: 652 [ 7192.057068] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 7192.063596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7192.097750] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7192.103671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7192.106508] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7192.108611] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7192.115958] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7192.118667] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7192.121077] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7192.125319] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7192.129373] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7192.136383] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7192.140553] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7192.144284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7192.146350] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7192.148964] Call Trace: [ 7192.150018] [ 7192.150755] ? show_regs.cold.9+0x22/0x2f [ 7192.152066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7192.154069] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7192.156090] ? watchdog+0x40/0x40 [ 7192.157670] ? hrtimer_interrupt+0x128/0x2c0 [ 7192.160001] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7192.162444] ? apic_timer_interrupt+0xf/0x20 [ 7192.164485] [ 7192.165513] ? native_safe_halt+0x2e/0x50 [ 7192.167399] kvm_wait+0xcd/0xe0 [ 7192.168477] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7192.170440] do_raw_spin_lock+0x177/0x190 [ 7192.172382] _raw_spin_lock+0x21/0x30 [ 7192.174415] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7192.177403] ? do_raw_spin_unlock+0x75/0x190 [ 7192.179465] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7192.181473] ? do_wait_intr+0xf0/0xf0 [ 7192.182666] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7192.184696] mdc_close+0x53b/0xd70 [mdc] [ 7192.186535] lmv_close+0x18d/0x490 [lmv] [ 7192.188571] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7192.191444] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7192.193739] ll_md_close+0x274/0x800 [lustre] [ 7192.196040] ll_file_release+0x128/0xc50 [lustre] [ 7192.198533] __fput+0x10d/0x440 [ 7192.200040] ____fput+0x12/0x20 [ 7192.204436] task_work_run+0xc6/0x110 [ 7192.205765] exit_to_usermode_loop+0x1dd/0x1f0 [ 7192.207446] do_syscall_64+0x3d6/0x440 [ 7192.212948] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7192.215932] RIP: 0033:0x7f1a63983cd8 [ 7192.218820] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7192.240280] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7192.248069] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7192.256263] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7192.264683] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7192.270406] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7192.279100] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7204.056028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 7204.058100] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 7204.066384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7204.066471] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7204.066479] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7204.066482] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7204.066495] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7204.066499] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7204.066504] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7204.066506] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7204.066508] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7204.066510] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7204.066513] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7204.066516] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7204.066519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7204.066522] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7204.066530] Call Trace: [ 7204.066535] [ 7204.066540] ? show_regs.cold.9+0x22/0x2f [ 7204.066550] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7204.066557] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7204.066560] ? watchdog+0x40/0x40 [ 7204.066565] ? hrtimer_interrupt+0x128/0x2c0 [ 7204.066569] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7204.072147] Modules linked in: lustre(O) [ 7204.114072] ? apic_timer_interrupt+0xf/0x20 [ 7204.114094] [ 7204.114100] ? native_safe_halt+0x2e/0x50 [ 7204.114105] kvm_wait+0xcd/0xe0 [ 7204.114115] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7204.114123] do_raw_spin_lock+0x177/0x190 [ 7204.114127] _raw_spin_lock+0x21/0x30 [ 7204.114132] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7204.114430] ? do_raw_spin_unlock+0x75/0x190 [ 7204.114437] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7204.114632] ? do_wait_intr+0xf0/0xf0 [ 7204.114639] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7204.122480] osp(O) [ 7204.126750] mdc_close+0x53b/0xd70 [mdc] [ 7204.129582] ofd(O) [ 7204.147150] lmv_close+0x18d/0x490 [lmv] [ 7204.151366] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [ 7204.161864] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7204.162001] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7204.165827] mdc(O) [ 7204.171690] ll_md_close+0x274/0x800 [lustre] [ 7204.178509] lov(O) [ 7204.181910] ll_file_release+0x128/0xc50 [lustre] [ 7204.187616] osc(O) [ 7204.199744] __fput+0x10d/0x440 [ 7204.199764] ____fput+0x12/0x20 [ 7204.199769] task_work_run+0xc6/0x110 [ 7204.199776] exit_to_usermode_loop+0x1dd/0x1f0 [ 7204.199783] do_syscall_64+0x3d6/0x440 [ 7204.199788] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7204.199796] RIP: 0033:0x7f1a63983cd8 [ 7204.199804] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7204.199808] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7204.199814] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7204.199816] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7204.199818] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7204.199820] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7204.199822] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7204.328433] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7204.352404] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7204.358591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7204.365414] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7204.368925] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7204.379534] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7204.383385] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7204.387413] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7204.392613] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7204.399828] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7204.404423] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7204.411421] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7204.416698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7204.419945] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7204.423443] Call Trace: [ 7204.424603] [ 7204.425406] ? show_regs.cold.9+0x22/0x2f [ 7204.426787] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7204.429080] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7204.431471] ? watchdog+0x40/0x40 [ 7204.433467] ? hrtimer_interrupt+0x128/0x2c0 [ 7204.437280] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7204.440362] ? apic_timer_interrupt+0xf/0x20 [ 7204.442137] [ 7204.443059] ? native_safe_halt+0x2e/0x50 [ 7204.444876] kvm_wait+0xcd/0xe0 [ 7204.446563] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7204.448814] do_raw_spin_lock+0x177/0x190 [ 7204.450471] _raw_spin_lock+0x21/0x30 [ 7204.452021] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7204.453900] obd_mod_free+0x46/0x140 [ptlrpc] [ 7204.456344] mdc_commit_open+0x91/0xb0 [mdc] [ 7204.457622] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7204.459468] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7204.461742] after_reply+0xe2c/0x1780 [ptlrpc] [ 7204.463516] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7204.465721] ? __next_timer_interrupt+0x160/0x160 [ 7204.467448] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7204.469210] ? do_wait_intr+0xf0/0xf0 [ 7204.470579] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7204.472604] kthread+0x1d1/0x200 [ 7204.473594] ? set_kthread_struct+0x70/0x70 [ 7204.474780] ret_from_fork+0x1f/0x30 [ 7208.053085] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 7208.057240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7208.076577] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7208.081167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7208.083601] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7208.084910] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7208.092899] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7208.096619] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7208.100864] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7208.104534] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7208.109066] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7208.113109] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7208.115692] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7208.119367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7208.123075] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7208.127849] Call Trace: [ 7208.129105] [ 7208.130289] ? show_regs.cold.9+0x22/0x2f [ 7208.133172] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7208.137601] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7208.139812] ? watchdog+0x40/0x40 [ 7208.142643] ? hrtimer_interrupt+0x128/0x2c0 [ 7208.150568] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7208.153984] ? apic_timer_interrupt+0xf/0x20 [ 7208.156793] [ 7208.157823] ? native_safe_halt+0x2e/0x50 [ 7208.159929] kvm_wait+0xcd/0xe0 [ 7208.161476] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7208.165592] do_raw_spin_lock+0x177/0x190 [ 7208.167793] _raw_spin_lock+0x21/0x30 [ 7208.169802] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7208.172573] ? do_raw_spin_unlock+0x75/0x190 [ 7208.174800] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7208.178390] ? do_wait_intr+0xf0/0xf0 [ 7208.179842] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7208.182886] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7208.186899] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7208.189646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7208.192384] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7208.194861] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7208.197199] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7208.200095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7208.202921] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7208.207597] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7208.210954] lmv_intent_open+0x338/0x1150 [lmv] [ 7208.213460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7208.218519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7208.222827] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7208.226133] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7208.228658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7208.231398] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7208.233251] ? do_raw_spin_unlock+0x75/0x190 [ 7208.235620] ? _raw_spin_unlock+0x12/0x30 [ 7208.237214] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7208.240223] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7208.242838] ll_atomic_open+0x36a/0xd70 [lustre] [ 7208.246476] lookup_open+0x356/0xb40 [ 7208.248494] do_last+0x313/0xfc0 [ 7208.249746] ? nd_jump_root+0xe5/0x160 [ 7208.250848] ? path_init+0x437/0x520 [ 7208.252862] path_openat+0xf7/0x500 [ 7208.254388] ? do_raw_spin_unlock+0x75/0x190 [ 7208.255985] do_filp_open+0x99/0x140 [ 7208.258100] ? getname_flags+0x6e/0x330 [ 7208.260610] ? __check_object_size+0xff/0x256 [ 7208.262821] ? do_raw_spin_unlock+0x75/0x190 [ 7208.264385] ? _raw_spin_unlock+0x12/0x30 [ 7208.265896] do_sys_openat2+0x2b4/0x410 [ 7208.267402] do_sys_open+0x73/0xa0 [ 7208.268772] __x64_sys_openat+0x24/0x30 [ 7208.270218] do_syscall_64+0xc1/0x440 [ 7208.274541] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7208.277852] RIP: 0033:0x7f1a63983332 [ 7208.280634] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7208.291255] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7208.298093] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7208.301736] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7208.305581] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7208.308735] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7208.312577] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7213.546084] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5190s! [ 7213.552720] Showing busy workqueues and worker pools: [ 7213.556341] workqueue events: flags=0x0 [ 7213.560023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7213.566273] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7213.572251] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7213.580586] pending: sec_gc_main [ptlrpc] [ 7213.586571] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7213.591560] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7213.603555] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7213.610366] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7213.618361] workqueue events_long: flags=0x0 [ 7213.622370] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7213.635919] pending: lu_objects_flush [obdclass] [ 7213.647057] workqueue events_unbound: flags=0x2 [ 7213.651675] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7213.658184] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7213.668323] workqueue events_power_efficient: flags=0x80 [ 7213.672806] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7213.675877] pending: neigh_periodic_work, do_cache_clean [ 7213.682665] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7213.689330] pending: neigh_periodic_work, check_lifetime [ 7213.693891] workqueue mm_percpu_wq: flags=0x8 [ 7213.696720] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7213.702253] pending: vmstat_update [ 7213.706273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7213.720178] pending: vmstat_update [ 7213.726897] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7213.736067] pending: vmstat_update [ 7213.738561] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7213.741841] pending: vmstat_update [ 7213.743568] workqueue writeback: flags=0x4a [ 7213.745671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7213.748078] pending: wb_workfn, wb_workfn [ 7213.754791] workqueue xprtiod: flags=0xa [ 7213.756803] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7213.765218] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7213.770927] workqueue ipv6_addrconf: flags=0x40008 [ 7213.773817] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7213.777295] pending: addrconf_verify_work [ 7213.780342] workqueue knbd0-recv: flags=0x1a [ 7213.781840] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7213.784899] in-flight: 647:recv_work [ 7213.788713] workqueue ptlrpc_pinger: flags=0x2 [ 7213.793591] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7213.796570] pending: ptlrpc_pinger_main [ptlrpc] [ 7213.801888] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7203s workers=2 idle: 652 [ 7220.058961] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 7220.066331] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7220.099426] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7220.109109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7220.114754] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.120787] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7220.133523] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7220.138289] RAX: 00000000000075c6 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7220.142726] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7220.151714] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7220.156443] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7220.160338] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7220.165314] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7220.169909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7220.173611] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7220.179201] Call Trace: [ 7220.182611] [ 7220.184138] ? show_regs.cold.9+0x22/0x2f [ 7220.190732] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7220.195452] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7220.202197] ? watchdog+0x40/0x40 [ 7220.205642] ? hrtimer_interrupt+0x128/0x2c0 [ 7220.207154] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7220.209716] ? apic_timer_interrupt+0xf/0x20 [ 7220.212963] [ 7220.213719] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.219903] do_raw_spin_lock+0x177/0x190 [ 7220.224713] _raw_spin_lock+0x21/0x30 [ 7220.226403] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7220.230764] ? do_raw_spin_unlock+0x75/0x190 [ 7220.238860] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7220.242419] ? do_wait_intr+0xf0/0xf0 [ 7220.245519] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7220.247572] mdc_close+0x53b/0xd70 [mdc] [ 7220.252195] lmv_close+0x18d/0x490 [lmv] [ 7220.254385] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7220.259202] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7220.262110] ll_md_close+0x274/0x800 [lustre] [ 7220.265749] ll_file_release+0x128/0xc50 [lustre] [ 7220.268367] __fput+0x10d/0x440 [ 7220.269792] ____fput+0x12/0x20 [ 7220.271198] task_work_run+0xc6/0x110 [ 7220.273030] exit_to_usermode_loop+0x1dd/0x1f0 [ 7220.275081] do_syscall_64+0x3d6/0x440 [ 7220.276796] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7220.279440] RIP: 0033:0x7f1a63983cd8 [ 7220.281149] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7220.299202] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7220.302461] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7220.306073] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7220.309262] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7220.311907] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7220.313858] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7232.056151] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7232.058114] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7232.059669] Modules linked in: lustre(O) [ 7232.063299] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7232.063425] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7232.063432] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7232.063435] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7232.063449] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7232.063453] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7232.063458] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7232.063460] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7232.063463] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7232.063465] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7232.063467] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7232.063470] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7232.063472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7232.063474] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7232.063482] Call Trace: [ 7232.063487] [ 7232.063493] ? show_regs.cold.9+0x22/0x2f [ 7232.063503] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7232.063509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7232.063513] ? watchdog+0x40/0x40 [ 7232.063518] ? hrtimer_interrupt+0x128/0x2c0 [ 7232.063521] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7232.063527] ? apic_timer_interrupt+0xf/0x20 [ 7232.063532] [ 7232.063535] ? native_safe_halt+0x2e/0x50 [ 7232.063538] kvm_wait+0xcd/0xe0 [ 7232.063545] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7232.063552] do_raw_spin_lock+0x177/0x190 [ 7232.063557] _raw_spin_lock+0x21/0x30 [ 7232.063562] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7232.063850] obd_mod_free+0x46/0x140 [ptlrpc] [ 7232.064101] mdc_commit_open+0x91/0xb0 [mdc] [ 7232.064141] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7232.067193] osp(O) [ 7232.092252] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7232.092535] after_reply+0xe2c/0x1780 [ptlrpc] [ 7232.092743] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7232.093171] ? __next_timer_interrupt+0x160/0x160 [ 7232.093182] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7232.093450] ? do_wait_intr+0xf0/0xf0 [ 7232.093461] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7232.107186] ofd(O) [ 7232.111364] kthread+0x1d1/0x200 [ 7232.112996] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [ 7232.121787] ? set_kthread_struct+0x70/0x70 [ 7232.121808] ret_from_fork+0x1f/0x30 [ 7232.125290] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7232.125357] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7232.125366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7232.125369] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7232.125382] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7232.125386] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7232.125391] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7232.125393] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7232.125396] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7232.125398] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7232.125401] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7232.125404] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7232.125408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7232.125410] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7232.125420] Call Trace: [ 7232.125425] [ 7232.125430] ? show_regs.cold.9+0x22/0x2f [ 7232.125441] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7232.459613] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7232.466351] ? watchdog+0x40/0x40 [ 7232.468973] ? hrtimer_interrupt+0x128/0x2c0 [ 7232.475314] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7232.479172] ? apic_timer_interrupt+0xf/0x20 [ 7232.481295] [ 7232.482928] ? native_safe_halt+0x2e/0x50 [ 7232.484185] kvm_wait+0xcd/0xe0 [ 7232.486755] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7232.490003] do_raw_spin_lock+0x177/0x190 [ 7232.492883] _raw_spin_lock+0x21/0x30 [ 7232.497465] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7232.502434] ? do_raw_spin_unlock+0x75/0x190 [ 7232.509446] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7232.511592] ? do_wait_intr+0xf0/0xf0 [ 7232.517419] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7232.520988] mdc_close+0x53b/0xd70 [mdc] [ 7232.529025] lmv_close+0x18d/0x490 [lmv] [ 7232.531232] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7232.537200] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7232.540863] ll_md_close+0x274/0x800 [lustre] [ 7232.543420] ll_file_release+0x128/0xc50 [lustre] [ 7232.546087] __fput+0x10d/0x440 [ 7232.547399] ____fput+0x12/0x20 [ 7232.549288] task_work_run+0xc6/0x110 [ 7232.550965] exit_to_usermode_loop+0x1dd/0x1f0 [ 7232.553034] do_syscall_64+0x3d6/0x440 [ 7232.555007] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7232.557274] RIP: 0033:0x7f1a63983cd8 [ 7232.558855] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7232.567372] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7232.579376] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7232.587186] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7232.598485] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7232.609713] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7232.617839] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7240.053066] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7240.056154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7240.084974] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7240.095795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7240.099728] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7240.104213] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7240.121403] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7240.125094] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7240.129667] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7240.133528] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7240.144827] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7240.148868] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7240.153277] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7240.158663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7240.162228] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7240.166156] Call Trace: [ 7240.167325] [ 7240.170096] ? show_regs.cold.9+0x22/0x2f [ 7240.174397] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7240.178945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7240.181807] ? watchdog+0x40/0x40 [ 7240.183892] ? hrtimer_interrupt+0x128/0x2c0 [ 7240.186480] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7240.189924] ? apic_timer_interrupt+0xf/0x20 [ 7240.192266] [ 7240.193323] ? native_safe_halt+0x2e/0x50 [ 7240.195194] kvm_wait+0xcd/0xe0 [ 7240.197219] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7240.199651] do_raw_spin_lock+0x177/0x190 [ 7240.201561] _raw_spin_lock+0x21/0x30 [ 7240.203619] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7240.206192] ? do_raw_spin_unlock+0x75/0x190 [ 7240.207942] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7240.210273] ? do_wait_intr+0xf0/0xf0 [ 7240.211553] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7240.213807] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7240.216793] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7240.220643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7240.226713] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7240.232504] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7240.236271] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7240.239003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7240.243440] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7240.248999] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7240.251359] lmv_intent_open+0x338/0x1150 [lmv] [ 7240.253347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7240.257525] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7240.262206] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7240.266609] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7240.270500] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7240.276763] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7240.280092] ? do_raw_spin_unlock+0x75/0x190 [ 7240.281559] ? _raw_spin_unlock+0x12/0x30 [ 7240.284035] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7240.286540] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7240.291118] ll_atomic_open+0x36a/0xd70 [lustre] [ 7240.295182] lookup_open+0x356/0xb40 [ 7240.299107] do_last+0x313/0xfc0 [ 7240.300901] ? nd_jump_root+0xe5/0x160 [ 7240.302327] ? path_init+0x437/0x520 [ 7240.304160] path_openat+0xf7/0x500 [ 7240.305507] ? do_raw_spin_unlock+0x75/0x190 [ 7240.307388] do_filp_open+0x99/0x140 [ 7240.308920] ? getname_flags+0x6e/0x330 [ 7240.310481] ? __check_object_size+0xff/0x256 [ 7240.312116] ? do_raw_spin_unlock+0x75/0x190 [ 7240.313864] ? _raw_spin_unlock+0x12/0x30 [ 7240.316209] do_sys_openat2+0x2b4/0x410 [ 7240.317758] do_sys_open+0x73/0xa0 [ 7240.319026] __x64_sys_openat+0x24/0x30 [ 7240.320518] do_syscall_64+0xc1/0x440 [ 7240.322085] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7240.324222] RIP: 0033:0x7f1a63983332 [ 7240.325668] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7240.337133] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7240.344249] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7240.348253] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7240.355270] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7240.359052] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7240.365993] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7244.257041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5221s! [ 7244.268763] Showing busy workqueues and worker pools: [ 7244.274854] workqueue events: flags=0x0 [ 7244.279395] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7244.287698] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7244.301984] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7244.315573] pending: sec_gc_main [ptlrpc] [ 7244.319381] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7244.323790] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7244.337580] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7244.345342] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7244.360448] workqueue events_long: flags=0x0 [ 7244.363250] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7244.366485] pending: lu_objects_flush [obdclass] [ 7244.372794] workqueue events_unbound: flags=0x2 [ 7244.378102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7244.383644] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7244.391758] workqueue events_power_efficient: flags=0x80 [ 7244.395979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7244.399127] pending: neigh_periodic_work, do_cache_clean [ 7244.402293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7244.416500] pending: neigh_periodic_work, check_lifetime [ 7244.420817] workqueue mm_percpu_wq: flags=0x8 [ 7244.422502] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7244.426010] pending: vmstat_update [ 7244.429513] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7244.435755] pending: vmstat_update [ 7244.438311] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7244.445101] pending: vmstat_update [ 7244.447044] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7244.450822] pending: vmstat_update [ 7244.452917] workqueue writeback: flags=0x4a [ 7244.456430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7244.460482] pending: wb_workfn, wb_workfn [ 7244.463193] workqueue xprtiod: flags=0xa [ 7244.465465] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7244.469210] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7244.473934] workqueue ipv6_addrconf: flags=0x40008 [ 7244.476388] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7244.480146] pending: addrconf_verify_work [ 7244.482179] workqueue knbd0-recv: flags=0x1a [ 7244.484169] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7244.488296] in-flight: 647:recv_work [ 7244.490088] workqueue ptlrpc_pinger: flags=0x2 [ 7244.492214] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7244.494607] pending: ptlrpc_pinger_main [ptlrpc] [ 7244.497472] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7234s workers=2 idle: 652 [ 7248.066766] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 7248.079023] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7248.118057] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7248.123393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7248.126728] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7248.128339] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7248.136282] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7248.139926] RAX: 0000000000007e99 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7248.142295] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7248.147507] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7248.153909] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7248.158578] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7248.162650] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7248.167209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7248.169816] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7248.173480] Call Trace: [ 7248.174795] [ 7248.175905] ? show_regs.cold.9+0x22/0x2f [ 7248.178954] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7248.181745] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7248.184906] ? watchdog+0x40/0x40 [ 7248.187466] ? hrtimer_interrupt+0x128/0x2c0 [ 7248.189591] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7248.192454] ? apic_timer_interrupt+0xf/0x20 [ 7248.195488] [ 7248.196528] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7248.199442] do_raw_spin_lock+0x177/0x190 [ 7248.201588] _raw_spin_lock+0x21/0x30 [ 7248.203591] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7248.206799] ? do_raw_spin_unlock+0x75/0x190 [ 7248.209345] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7248.212500] ? do_wait_intr+0xf0/0xf0 [ 7248.214495] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7248.217395] mdc_close+0x53b/0xd70 [mdc] [ 7248.219349] lmv_close+0x18d/0x490 [lmv] [ 7248.221057] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7248.223317] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7248.225714] ll_md_close+0x274/0x800 [lustre] [ 7248.227765] ll_file_release+0x128/0xc50 [lustre] [ 7248.229605] __fput+0x10d/0x440 [ 7248.230802] ____fput+0x12/0x20 [ 7248.231968] task_work_run+0xc6/0x110 [ 7248.233351] exit_to_usermode_loop+0x1dd/0x1f0 [ 7248.235264] do_syscall_64+0x3d6/0x440 [ 7248.236889] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7248.239043] RIP: 0033:0x7f1a63983cd8 [ 7248.240427] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7248.248209] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7248.251369] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7248.253707] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7248.256602] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7248.259937] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7248.262275] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7260.056013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7260.058008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7260.059808] Modules linked in: lustre(O) [ 7260.064631] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7260.064710] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7260.064718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7260.064721] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7260.064734] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7260.064737] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7260.064741] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7260.064743] RDX: 00000000000052c7 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7260.064745] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7260.064747] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7260.064749] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7260.064752] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7260.064754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7260.064756] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7260.064765] Call Trace: [ 7260.064771] [ 7260.064778] ? show_regs.cold.9+0x22/0x2f [ 7260.064788] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7260.064794] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7260.064799] ? watchdog+0x40/0x40 [ 7260.064803] ? hrtimer_interrupt+0x128/0x2c0 [ 7260.064807] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7260.064816] ? apic_timer_interrupt+0xf/0x20 [ 7260.064822] [ 7260.064825] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7260.064831] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7260.065154] do_raw_spin_lock+0x177/0x190 [ 7260.066047] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7260.066089] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7260.066093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7260.066095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7260.066102] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7260.066105] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7260.066107] RAX: 000000000000450a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7260.066109] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7260.066114] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7260.066115] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7260.066117] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7260.066119] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7260.066121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7260.066122] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7260.066127] Call Trace: [ 7260.066131] [ 7260.066135] ? show_regs.cold.9+0x22/0x2f [ 7260.066140] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7260.066143] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7260.066145] ? watchdog+0x40/0x40 [ 7260.066148] ? hrtimer_interrupt+0x128/0x2c0 [ 7260.066151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7260.066156] ? apic_timer_interrupt+0xf/0x20 [ 7260.066160] [ 7260.066161] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7260.066165] do_raw_spin_lock+0x177/0x190 [ 7260.066168] _raw_spin_lock+0x21/0x30 [ 7260.066172] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7260.066445] ? do_raw_spin_unlock+0x75/0x190 [ 7260.066449] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7260.066728] ? do_wait_intr+0xf0/0xf0 [ 7260.066736] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7260.066927] mdc_close+0x53b/0xd70 [mdc] [ 7260.066964] lmv_close+0x18d/0x490 [lmv] [ 7260.067315] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7260.067436] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7260.067696] ll_md_close+0x274/0x800 [lustre] [ 7260.067783] ll_file_release+0x128/0xc50 [lustre] [ 7260.067866] __fput+0x10d/0x440 [ 7260.067875] ____fput+0x12/0x20 [ 7260.067879] task_work_run+0xc6/0x110 [ 7260.067887] exit_to_usermode_loop+0x1dd/0x1f0 [ 7260.067894] do_syscall_64+0x3d6/0x440 [ 7260.067898] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7260.067906] RIP: 0033:0x7f1a63983cd8 [ 7260.067914] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7260.067917] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7260.067922] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7260.067925] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7260.067927] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7260.067930] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7260.067932] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7260.284288] _raw_spin_lock+0x21/0x30 [ 7260.285827] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7260.288222] obd_mod_free+0x46/0x140 [ptlrpc] [ 7260.290206] mdc_commit_open+0x91/0xb0 [mdc] [ 7260.291899] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7260.294280] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7260.296584] after_reply+0xe2c/0x1780 [ptlrpc] [ 7260.298713] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7260.300985] ? __next_timer_interrupt+0x160/0x160 [ 7260.302821] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7260.304785] ? do_wait_intr+0xf0/0xf0 [ 7260.306178] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7260.308459] kthread+0x1d1/0x200 [ 7260.309872] ? set_kthread_struct+0x70/0x70 [ 7260.311564] ret_from_fork+0x1f/0x30 [ 7272.053054] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7272.056785] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7272.086078] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7272.092378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7272.097413] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7272.099753] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7272.109844] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7272.114188] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7272.120924] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7272.127011] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7272.131503] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7272.135591] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7272.139534] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7272.144931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7272.149543] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7272.154808] Call Trace: [ 7272.156114] [ 7272.157291] ? show_regs.cold.9+0x22/0x2f [ 7272.159340] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7272.162050] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7272.164618] ? watchdog+0x40/0x40 [ 7272.167647] ? hrtimer_interrupt+0x128/0x2c0 [ 7272.169995] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7272.172512] ? apic_timer_interrupt+0xf/0x20 [ 7272.174674] [ 7272.175647] ? native_safe_halt+0x2e/0x50 [ 7272.177931] kvm_wait+0xcd/0xe0 [ 7272.179540] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7272.182672] do_raw_spin_lock+0x177/0x190 [ 7272.187178] _raw_spin_lock+0x21/0x30 [ 7272.189619] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7272.192152] ? do_raw_spin_unlock+0x75/0x190 [ 7272.194819] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7272.199884] ? do_wait_intr+0xf0/0xf0 [ 7272.204785] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7272.208548] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7272.211955] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7272.214865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7272.219581] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7272.222324] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7272.230380] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7272.233042] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7272.235929] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7272.238635] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7272.240373] lmv_intent_open+0x338/0x1150 [lmv] [ 7272.242331] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7272.247326] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7272.250062] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7272.253810] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7272.256893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7272.262075] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7272.264909] ? do_raw_spin_unlock+0x75/0x190 [ 7272.266915] ? _raw_spin_unlock+0x12/0x30 [ 7272.269063] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7272.273451] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7272.276897] ll_atomic_open+0x36a/0xd70 [lustre] [ 7272.279501] lookup_open+0x356/0xb40 [ 7272.281376] do_last+0x313/0xfc0 [ 7272.282890] ? nd_jump_root+0xe5/0x160 [ 7272.285277] ? path_init+0x437/0x520 [ 7272.286972] path_openat+0xf7/0x500 [ 7272.288935] ? do_raw_spin_unlock+0x75/0x190 [ 7272.290628] do_filp_open+0x99/0x140 [ 7272.292680] ? getname_flags+0x6e/0x330 [ 7272.294649] ? __check_object_size+0xff/0x256 [ 7272.296600] ? do_raw_spin_unlock+0x75/0x190 [ 7272.299117] ? _raw_spin_unlock+0x12/0x30 [ 7272.303777] do_sys_openat2+0x2b4/0x410 [ 7272.305259] do_sys_open+0x73/0xa0 [ 7272.306630] __x64_sys_openat+0x24/0x30 [ 7272.308366] do_syscall_64+0xc1/0x440 [ 7272.311603] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7272.315035] RIP: 0033:0x7f1a63983332 [ 7272.317771] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7272.325613] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7272.330153] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7272.342601] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7272.349460] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7272.352769] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7272.356639] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7274.978092] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5251s! [ 7274.990927] Showing busy workqueues and worker pools: [ 7274.998117] workqueue events: flags=0x0 [ 7275.004025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7275.013264] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7275.026401] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7275.032312] pending: sec_gc_main [ptlrpc] [ 7275.040185] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7275.049819] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7275.066907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7275.071088] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7275.083020] workqueue events_long: flags=0x0 [ 7275.086189] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7275.090084] pending: lu_objects_flush [obdclass] [ 7275.094242] workqueue events_unbound: flags=0x2 [ 7275.096783] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7275.099875] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7275.103323] workqueue events_power_efficient: flags=0x80 [ 7275.105526] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7275.109215] pending: neigh_periodic_work, do_cache_clean [ 7275.111479] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7275.114257] pending: neigh_periodic_work, check_lifetime [ 7275.117762] workqueue mm_percpu_wq: flags=0x8 [ 7275.120208] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7275.123645] pending: vmstat_update [ 7275.125443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7275.128803] pending: vmstat_update [ 7275.130031] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7275.132654] pending: vmstat_update [ 7275.134093] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7275.137259] pending: vmstat_update [ 7275.139053] workqueue writeback: flags=0x4a [ 7275.140655] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7275.143255] pending: wb_workfn, wb_workfn [ 7275.145851] workqueue xprtiod: flags=0xa [ 7275.147483] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7275.150049] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7275.153838] workqueue ipv6_addrconf: flags=0x40008 [ 7275.155187] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7275.157574] pending: addrconf_verify_work [ 7275.158828] workqueue knbd0-recv: flags=0x1a [ 7275.160018] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7275.162066] in-flight: 647:recv_work [ 7275.163158] workqueue ptlrpc_pinger: flags=0x2 [ 7275.164487] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7275.166117] pending: ptlrpc_pinger_main [ptlrpc] [ 7275.167865] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7265s workers=2 idle: 652 [ 7276.057031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7276.060530] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7276.084707] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7276.089894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7276.093394] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7276.096193] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7276.104336] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7276.107923] RAX: 000000000000590b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7276.111711] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7276.115296] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7276.118636] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7276.121814] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7276.124631] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7276.127794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7276.131045] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7276.134637] Call Trace: [ 7276.135965] [ 7276.137304] ? show_regs.cold.9+0x22/0x2f [ 7276.140175] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7276.148330] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7276.152339] ? watchdog+0x40/0x40 [ 7276.156370] ? hrtimer_interrupt+0x128/0x2c0 [ 7276.160148] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7276.162364] ? apic_timer_interrupt+0xf/0x20 [ 7276.171482] [ 7276.174489] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7276.179992] do_raw_spin_lock+0x177/0x190 [ 7276.182304] _raw_spin_lock+0x21/0x30 [ 7276.184102] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7276.187467] ? do_raw_spin_unlock+0x75/0x190 [ 7276.191317] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7276.200800] ? do_wait_intr+0xf0/0xf0 [ 7276.202304] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7276.209151] mdc_close+0x53b/0xd70 [mdc] [ 7276.210784] lmv_close+0x18d/0x490 [lmv] [ 7276.212302] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7276.215042] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7276.217164] ll_md_close+0x274/0x800 [lustre] [ 7276.219109] ll_file_release+0x128/0xc50 [lustre] [ 7276.221820] __fput+0x10d/0x440 [ 7276.223147] ____fput+0x12/0x20 [ 7276.224768] task_work_run+0xc6/0x110 [ 7276.226693] exit_to_usermode_loop+0x1dd/0x1f0 [ 7276.229263] do_syscall_64+0x3d6/0x440 [ 7276.230906] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7276.233245] RIP: 0033:0x7f1a63983cd8 [ 7276.234954] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7276.251626] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7276.256604] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7276.261957] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7276.269672] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7276.275649] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7276.278721] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7288.056029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7288.058008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7288.060921] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7288.060999] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7288.061006] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7288.061009] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7288.061024] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7288.061029] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7288.061034] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7288.061037] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7288.061040] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7288.065122] Modules linked in: lustre(O) [ 7288.097046] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7288.104818] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7288.104899] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7288.104906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7288.104909] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7288.104923] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7288.104927] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7288.104932] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7288.104935] RDX: 00000000000000fb RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7288.104937] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7288.104940] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7288.104943] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7288.104946] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7288.104949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7288.104951] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7288.104960] Call Trace: [ 7288.104964] [ 7288.104969] ? show_regs.cold.9+0x22/0x2f [ 7288.104977] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7288.104983] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7288.104987] ? watchdog+0x40/0x40 [ 7288.104992] ? hrtimer_interrupt+0x128/0x2c0 [ 7288.104996] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7288.105004] ? apic_timer_interrupt+0xf/0x20 [ 7288.105008] [ 7288.105011] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7288.105016] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7288.105022] do_raw_spin_lock+0x177/0x190 [ 7288.105027] _raw_spin_lock+0x21/0x30 [ 7288.105033] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7288.105445] obd_mod_free+0x46/0x140 [ptlrpc] [ 7288.105656] mdc_commit_open+0x91/0xb0 [mdc] [ 7288.105694] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7288.105892] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7288.106098] after_reply+0xe2c/0x1780 [ptlrpc] [ 7288.106572] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7288.106817] ? __next_timer_interrupt+0x160/0x160 [ 7288.106826] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7288.107069] ? do_wait_intr+0xf0/0xf0 [ 7288.107078] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7288.113458] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7288.122843] kthread+0x1d1/0x200 [ 7288.122863] ? set_kthread_struct+0x70/0x70 [ 7288.122867] ret_from_fork+0x1f/0x30 [ 7288.253429] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7288.256615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7288.259062] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7288.261877] Call Trace: [ 7288.263031] [ 7288.263914] ? show_regs.cold.9+0x22/0x2f [ 7288.265860] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7288.267949] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7288.269658] ? watchdog+0x40/0x40 [ 7288.270829] ? hrtimer_interrupt+0x128/0x2c0 [ 7288.272328] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7288.273752] ? apic_timer_interrupt+0xf/0x20 [ 7288.275604] [ 7288.276372] ? native_safe_halt+0x2e/0x50 [ 7288.277994] kvm_wait+0xcd/0xe0 [ 7288.279187] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7288.280949] do_raw_spin_lock+0x177/0x190 [ 7288.283309] _raw_spin_lock+0x21/0x30 [ 7288.285789] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7288.287646] ? do_raw_spin_unlock+0x75/0x190 [ 7288.289103] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7288.291111] ? do_wait_intr+0xf0/0xf0 [ 7288.293798] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7288.297703] mdc_close+0x53b/0xd70 [mdc] [ 7288.301821] lmv_close+0x18d/0x490 [lmv] [ 7288.303673] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7288.306942] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7288.310053] ll_md_close+0x274/0x800 [lustre] [ 7288.312480] ll_file_release+0x128/0xc50 [lustre] [ 7288.314664] __fput+0x10d/0x440 [ 7288.315810] ____fput+0x12/0x20 [ 7288.316885] task_work_run+0xc6/0x110 [ 7288.318091] exit_to_usermode_loop+0x1dd/0x1f0 [ 7288.321137] do_syscall_64+0x3d6/0x440 [ 7288.322429] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7288.326485] RIP: 0033:0x7f1a63983cd8 [ 7288.327767] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7288.341793] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7288.349454] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7288.357476] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7288.363973] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7288.369906] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7288.377732] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7300.053856] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7300.067713] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7300.107467] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7300.118911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7300.128083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7300.131466] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7300.151788] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7300.160082] RAX: 0000000000007e85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7300.171683] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7300.179103] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7300.183539] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7300.190385] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7300.194623] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7300.202069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7300.209197] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7300.213994] Call Trace: [ 7300.216098] [ 7300.222177] ? show_regs.cold.9+0x22/0x2f [ 7300.224379] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7300.226687] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7300.232182] ? watchdog+0x40/0x40 [ 7300.233464] ? hrtimer_interrupt+0x128/0x2c0 [ 7300.239671] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7300.243229] ? apic_timer_interrupt+0xf/0x20 [ 7300.246795] [ 7300.248302] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7300.252712] do_raw_spin_lock+0x177/0x190 [ 7300.256536] _raw_spin_lock+0x21/0x30 [ 7300.258583] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7300.261260] ? do_raw_spin_unlock+0x75/0x190 [ 7300.262918] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7300.265695] ? do_wait_intr+0xf0/0xf0 [ 7300.267493] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7300.271973] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7300.277467] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7300.281689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7300.284354] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7300.291321] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7300.300206] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7300.301694] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7300.319736] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7300.330091] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7300.335455] lmv_intent_open+0x338/0x1150 [lmv] [ 7300.341445] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7300.344072] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7300.347183] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7300.352358] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7300.358736] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7300.363423] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7300.365861] ? do_raw_spin_unlock+0x75/0x190 [ 7300.367052] ? _raw_spin_unlock+0x12/0x30 [ 7300.385328] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7300.386810] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7300.393522] ll_atomic_open+0x36a/0xd70 [lustre] [ 7300.395359] lookup_open+0x356/0xb40 [ 7300.405374] do_last+0x313/0xfc0 [ 7300.406733] ? nd_jump_root+0xe5/0x160 [ 7300.408033] ? path_init+0x437/0x520 [ 7300.410762] path_openat+0xf7/0x500 [ 7300.412725] ? do_raw_spin_unlock+0x75/0x190 [ 7300.415590] do_filp_open+0x99/0x140 [ 7300.420912] ? getname_flags+0x6e/0x330 [ 7300.422987] ? __check_object_size+0xff/0x256 [ 7300.433267] ? do_raw_spin_unlock+0x75/0x190 [ 7300.434950] ? _raw_spin_unlock+0x12/0x30 [ 7300.444984] do_sys_openat2+0x2b4/0x410 [ 7300.448496] do_sys_open+0x73/0xa0 [ 7300.452273] __x64_sys_openat+0x24/0x30 [ 7300.454399] do_syscall_64+0xc1/0x440 [ 7300.457353] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7300.459770] RIP: 0033:0x7f1a63983332 [ 7300.462708] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7300.474965] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7300.478335] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7300.480990] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7300.486621] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7300.489992] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7300.494015] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7302.529058] rcu: INFO: rcu_sched self-detected stall on CPU [ 7302.533388] rcu: 3-...!: (3303296 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 7302.540830] (t=5280134 jiffies g=269633 q=1908) [ 7302.543628] rcu: rcu_sched kthread starved for 5279783 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 7302.559121] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7302.569851] rcu: RCU grace-period kthread stack dump: [ 7302.571916] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7302.576370] Call Trace: [ 7302.578431] __schedule+0x351/0xcb0 [ 7302.579755] schedule+0xc0/0x180 [ 7302.581887] schedule_timeout+0xb4/0x190 [ 7302.584187] ? do_raw_spin_unlock+0x75/0x190 [ 7302.585907] ? __next_timer_interrupt+0x160/0x160 [ 7302.587923] ? prepare_to_swait_event+0xf3/0x140 [ 7302.589952] rcu_gp_kthread+0x8dd/0x15b0 [ 7302.591295] ? force_qs_rnp+0x2c0/0x2c0 [ 7302.592532] kthread+0x1d1/0x200 [ 7302.594248] ? set_kthread_struct+0x70/0x70 [ 7302.597007] ret_from_fork+0x1f/0x30 [ 7302.598493] rcu: Stack dump where RCU GP kthread last ran: [ 7302.600461] Sending NMI from CPU 3 to CPUs 0: [ 7302.606878] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 7302.612543] NMI backtrace for cpu 3 [ 7302.615817] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7302.629609] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7302.647423] Call Trace: [ 7302.648930] [ 7302.649892] ? dump_stack+0xbb/0x10e [ 7302.653164] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7302.662605] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7302.664500] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7302.679425] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7302.691382] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7302.699351] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7302.704897] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7302.709903] ? update_process_times+0x6d/0xc0 [ 7302.715496] ? tick_sched_handle.isra.18+0x27/0x90 [ 7302.718744] ? tick_sched_timer+0x53/0xb0 [ 7302.720082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7302.726710] ? tick_sched_handle.isra.18+0x90/0x90 [ 7302.728516] ? hrtimer_interrupt+0x128/0x2c0 [ 7302.730354] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7302.732478] ? apic_timer_interrupt+0xf/0x20 [ 7302.734441] [ 7302.735285] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7302.737545] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7302.744621] ? do_raw_spin_lock+0x177/0x190 [ 7302.749578] ? _raw_spin_lock+0x21/0x30 [ 7302.753805] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7302.756760] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7302.762738] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7302.764465] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7302.775917] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7302.787962] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7302.792595] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7302.798405] ? __next_timer_interrupt+0x160/0x160 [ 7302.803425] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7302.808396] ? do_wait_intr+0xf0/0xf0 [ 7302.812716] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7302.815367] ? kthread+0x1d1/0x200 [ 7302.817908] ? set_kthread_struct+0x70/0x70 [ 7302.819827] ? ret_from_fork+0x1f/0x30 [ 7304.057139] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7304.061166] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7304.104968] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7304.117587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7304.126879] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7304.133595] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7304.153005] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7304.160137] RAX: 00000000000059ce RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7304.166832] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7304.174317] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7304.180209] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7304.184971] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7304.194942] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7304.199647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7304.209591] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7304.216498] Call Trace: [ 7304.220025] [ 7304.224048] ? show_regs.cold.9+0x22/0x2f [ 7304.226918] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7304.230610] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7304.235797] ? watchdog+0x40/0x40 [ 7304.237013] ? hrtimer_interrupt+0x128/0x2c0 [ 7304.240868] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7304.244211] ? apic_timer_interrupt+0xf/0x20 [ 7304.249472] [ 7304.250389] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7304.252588] do_raw_spin_lock+0x177/0x190 [ 7304.254407] _raw_spin_lock+0x21/0x30 [ 7304.256082] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7304.259190] ? do_raw_spin_unlock+0x75/0x190 [ 7304.261282] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7304.263909] ? do_wait_intr+0xf0/0xf0 [ 7304.265668] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7304.268478] mdc_close+0x53b/0xd70 [mdc] [ 7304.270584] lmv_close+0x18d/0x490 [lmv] [ 7304.272506] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7304.275049] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7304.277611] ll_md_close+0x274/0x800 [lustre] [ 7304.279748] ll_file_release+0x128/0xc50 [lustre] [ 7304.282125] __fput+0x10d/0x440 [ 7304.284249] ____fput+0x12/0x20 [ 7304.285859] task_work_run+0xc6/0x110 [ 7304.289603] exit_to_usermode_loop+0x1dd/0x1f0 [ 7304.292550] do_syscall_64+0x3d6/0x440 [ 7304.301420] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7304.308892] RIP: 0033:0x7f1a63983cd8 [ 7304.316815] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7304.346821] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7304.351755] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7304.362784] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7304.371654] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7304.380361] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7304.386411] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7305.185391] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5282s! [ 7305.192316] Showing busy workqueues and worker pools: [ 7305.194378] workqueue events: flags=0x0 [ 7305.195404] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7305.202934] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7305.206849] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7305.209807] pending: sec_gc_main [ptlrpc] [ 7305.212536] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7305.215779] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7305.220791] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7305.224302] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7305.229208] workqueue events_long: flags=0x0 [ 7305.230842] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7305.234607] pending: lu_objects_flush [obdclass] [ 7305.238222] workqueue events_unbound: flags=0x2 [ 7305.240667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7305.247208] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7305.254821] workqueue events_power_efficient: flags=0x80 [ 7305.258162] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7305.262304] pending: neigh_periodic_work, do_cache_clean [ 7305.267381] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7305.272725] pending: neigh_periodic_work, check_lifetime [ 7305.275606] workqueue mm_percpu_wq: flags=0x8 [ 7305.278307] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7305.282820] pending: vmstat_update [ 7305.288798] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7305.295589] pending: vmstat_update [ 7305.300522] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7305.306373] pending: vmstat_update [ 7305.308783] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7305.314162] pending: vmstat_update [ 7305.316531] workqueue writeback: flags=0x4a [ 7305.318872] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7305.322502] pending: wb_workfn, wb_workfn [ 7305.324768] workqueue xprtiod: flags=0xa [ 7305.327797] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7305.331267] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7305.335367] workqueue ipv6_addrconf: flags=0x40008 [ 7305.337792] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7305.341896] pending: addrconf_verify_work [ 7305.344285] workqueue knbd0-recv: flags=0x1a [ 7305.346388] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7305.350177] in-flight: 647:recv_work [ 7305.352752] workqueue ptlrpc_pinger: flags=0x2 [ 7305.355230] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7305.358041] pending: ptlrpc_pinger_main [ptlrpc] [ 7305.361306] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7295s workers=2 idle: 652 [ 7316.056044] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7316.059078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7316.089624] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7316.094294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7316.099720] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7316.101320] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7316.109925] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7316.117699] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7316.122553] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7316.125540] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7316.130526] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7316.137453] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7316.140190] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7316.142762] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7316.144683] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7316.147794] Call Trace: [ 7316.148636] [ 7316.149418] ? show_regs.cold.9+0x22/0x2f [ 7316.150600] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7316.153453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7316.154685] ? watchdog+0x40/0x40 [ 7316.155749] ? hrtimer_interrupt+0x128/0x2c0 [ 7316.157816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7316.161697] ? apic_timer_interrupt+0xf/0x20 [ 7316.164466] [ 7316.165183] ? native_safe_halt+0x2e/0x50 [ 7316.166486] kvm_wait+0xcd/0xe0 [ 7316.167420] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7316.170199] do_raw_spin_lock+0x177/0x190 [ 7316.172743] _raw_spin_lock+0x21/0x30 [ 7316.174783] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7316.176728] ? do_raw_spin_unlock+0x75/0x190 [ 7316.179159] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7316.182372] ? do_wait_intr+0xf0/0xf0 [ 7316.184195] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7316.186522] mdc_close+0x53b/0xd70 [mdc] [ 7316.189812] lmv_close+0x18d/0x490 [lmv] [ 7316.193730] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7316.195800] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7316.197491] ll_md_close+0x274/0x800 [lustre] [ 7316.199476] ll_file_release+0x128/0xc50 [lustre] [ 7316.201243] __fput+0x10d/0x440 [ 7316.202350] ____fput+0x12/0x20 [ 7316.203451] task_work_run+0xc6/0x110 [ 7316.206068] exit_to_usermode_loop+0x1dd/0x1f0 [ 7316.208806] do_syscall_64+0x3d6/0x440 [ 7316.210729] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7316.212514] RIP: 0033:0x7f1a63983cd8 [ 7316.213625] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7316.227944] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7316.230576] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7316.233067] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7316.235580] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7316.238384] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7316.240776] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7328.058365] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7328.069141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7328.112117] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7328.126323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7328.140626] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7328.148333] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7328.175612] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7328.184196] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7328.191461] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7328.197360] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7328.208732] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7328.213276] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7328.223698] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7328.226140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7328.227947] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7328.232027] Call Trace: [ 7328.233143] [ 7328.234162] ? show_regs.cold.9+0x22/0x2f [ 7328.236339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7328.240151] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7328.242507] ? watchdog+0x40/0x40 [ 7328.244164] ? hrtimer_interrupt+0x128/0x2c0 [ 7328.246235] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7328.248813] ? apic_timer_interrupt+0xf/0x20 [ 7328.251075] [ 7328.252125] ? native_safe_halt+0x2e/0x50 [ 7328.254127] kvm_wait+0xcd/0xe0 [ 7328.255830] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7328.258445] do_raw_spin_lock+0x177/0x190 [ 7328.260441] _raw_spin_lock+0x21/0x30 [ 7328.262353] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7328.265165] obd_mod_free+0x46/0x140 [ptlrpc] [ 7328.267616] mdc_commit_open+0x91/0xb0 [mdc] [ 7328.272764] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7328.277628] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7328.284257] after_reply+0xe2c/0x1780 [ptlrpc] [ 7328.288681] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7328.292316] ? __next_timer_interrupt+0x160/0x160 [ 7328.296050] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7328.298352] ? do_wait_intr+0xf0/0xf0 [ 7328.300258] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7328.302498] kthread+0x1d1/0x200 [ 7328.304082] ? set_kthread_struct+0x70/0x70 [ 7328.305920] ret_from_fork+0x1f/0x30 [ 7332.057094] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7332.057241] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7332.061042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7332.061129] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7332.061137] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7332.061140] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7332.061153] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7332.061157] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7332.061162] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7332.061165] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7332.061167] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7332.061169] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7332.061172] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7332.061175] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7332.061179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7332.061181] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7332.061189] Call Trace: [ 7332.061196] [ 7332.061201] ? show_regs.cold.9+0x22/0x2f [ 7332.061211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7332.061373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7332.061378] ? watchdog+0x40/0x40 [ 7332.061439] ? hrtimer_interrupt+0x128/0x2c0 [ 7332.061443] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7332.061449] ? apic_timer_interrupt+0xf/0x20 [ 7332.061453] [ 7332.061456] ? native_safe_halt+0x2e/0x50 [ 7332.061460] kvm_wait+0xcd/0xe0 [ 7332.061467] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7332.061475] do_raw_spin_lock+0x177/0x190 [ 7332.061478] _raw_spin_lock+0x21/0x30 [ 7332.061483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7332.061776] ? do_raw_spin_unlock+0x75/0x190 [ 7332.061783] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7332.062021] ? do_wait_intr+0xf0/0xf0 [ 7332.062029] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7332.062391] mdc_close+0x53b/0xd70 [mdc] [ 7332.062428] lmv_close+0x18d/0x490 [lmv] [ 7332.062467] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7332.062785] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7332.062895] ll_md_close+0x274/0x800 [lustre] [ 7332.062992] ll_file_release+0x128/0xc50 [lustre] [ 7332.063081] __fput+0x10d/0x440 [ 7332.063090] ____fput+0x12/0x20 [ 7332.063096] task_work_run+0xc6/0x110 [ 7332.063102] exit_to_usermode_loop+0x1dd/0x1f0 [ 7332.063109] do_syscall_64+0x3d6/0x440 [ 7332.063114] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7332.063122] RIP: 0033:0x7f1a63983cd8 [ 7332.063129] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7332.063133] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7332.063137] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7332.063140] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7332.063142] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7332.063144] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7332.063146] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7332.215092] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7332.237018] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7332.243608] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7332.248092] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.250474] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7332.264744] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7332.270308] RAX: 0000000000006e88 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7332.273312] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7332.276184] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7332.279438] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7332.296970] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7332.299318] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7332.306413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7332.311316] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7332.314548] Call Trace: [ 7332.315614] [ 7332.316713] ? show_regs.cold.9+0x22/0x2f [ 7332.318041] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7332.319725] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7332.322354] ? watchdog+0x40/0x40 [ 7332.324130] ? hrtimer_interrupt+0x128/0x2c0 [ 7332.325569] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7332.327937] ? apic_timer_interrupt+0xf/0x20 [ 7332.329533] [ 7332.330332] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.332394] do_raw_spin_lock+0x177/0x190 [ 7332.334109] _raw_spin_lock+0x21/0x30 [ 7332.335340] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7332.337753] ? do_raw_spin_unlock+0x75/0x190 [ 7332.339490] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7332.341603] ? do_wait_intr+0xf0/0xf0 [ 7332.343295] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7332.347262] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7332.349543] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7332.352410] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7332.355047] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7332.360977] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7332.363486] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7332.367463] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7332.371007] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7332.376422] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7332.383298] lmv_intent_open+0x338/0x1150 [lmv] [ 7332.385655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7332.389050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7332.392272] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7332.394505] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7332.397056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7332.400210] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7332.402996] ? do_raw_spin_unlock+0x75/0x190 [ 7332.405191] ? _raw_spin_unlock+0x12/0x30 [ 7332.407524] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7332.409653] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7332.414736] ll_atomic_open+0x36a/0xd70 [lustre] [ 7332.417168] lookup_open+0x356/0xb40 [ 7332.419131] do_last+0x313/0xfc0 [ 7332.420476] ? nd_jump_root+0xe5/0x160 [ 7332.422449] ? path_init+0x437/0x520 [ 7332.424040] path_openat+0xf7/0x500 [ 7332.425976] ? do_raw_spin_unlock+0x75/0x190 [ 7332.428665] do_filp_open+0x99/0x140 [ 7332.429949] ? getname_flags+0x6e/0x330 [ 7332.431353] ? __check_object_size+0xff/0x256 [ 7332.433242] ? do_raw_spin_unlock+0x75/0x190 [ 7332.435623] ? _raw_spin_unlock+0x12/0x30 [ 7332.437430] do_sys_openat2+0x2b4/0x410 [ 7332.439416] do_sys_open+0x73/0xa0 [ 7332.441009] __x64_sys_openat+0x24/0x30 [ 7332.442866] do_syscall_64+0xc1/0x440 [ 7332.444593] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7332.446913] RIP: 0033:0x7f1a63983332 [ 7332.449101] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7332.458136] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7332.461494] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7332.464927] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7332.468850] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7332.472170] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7332.476606] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7335.397926] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5312s! [ 7335.411558] Showing busy workqueues and worker pools: [ 7335.413360] workqueue events: flags=0x0 [ 7335.414682] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7335.428581] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7335.436653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7335.442561] pending: sec_gc_main [ptlrpc] [ 7335.447541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7335.452560] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7335.459209] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7335.472727] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7335.481614] workqueue events_long: flags=0x0 [ 7335.486594] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7335.501336] pending: lu_objects_flush [obdclass] [ 7335.506254] workqueue events_unbound: flags=0x2 [ 7335.507834] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7335.515186] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7335.526269] workqueue events_power_efficient: flags=0x80 [ 7335.527977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7335.533746] pending: neigh_periodic_work, do_cache_clean [ 7335.538589] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7335.548178] pending: neigh_periodic_work, check_lifetime [ 7335.555511] workqueue mm_percpu_wq: flags=0x8 [ 7335.557988] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7335.561608] pending: vmstat_update [ 7335.565586] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7335.568957] pending: vmstat_update [ 7335.571475] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7335.576657] pending: vmstat_update [ 7335.579082] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7335.586488] pending: vmstat_update [ 7335.588678] workqueue writeback: flags=0x4a [ 7335.591524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7335.603150] pending: wb_workfn, wb_workfn [ 7335.606910] workqueue xprtiod: flags=0xa [ 7335.609121] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7335.612509] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7335.618526] workqueue ipv6_addrconf: flags=0x40008 [ 7335.623463] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7335.631501] pending: addrconf_verify_work [ 7335.638174] workqueue knbd0-recv: flags=0x1a [ 7335.640635] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7335.643470] in-flight: 647:recv_work [ 7335.645144] workqueue ptlrpc_pinger: flags=0x2 [ 7335.647836] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7335.654628] pending: ptlrpc_pinger_main [ptlrpc] [ 7335.666452] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7325s workers=2 idle: 652 [ 7344.056654] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7344.066431] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7344.117255] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7344.139397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7344.147158] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7344.163200] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7344.181441] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7344.188926] RAX: 0000000000003782 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7344.198749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7344.207661] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7344.219126] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7344.224641] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7344.229228] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7344.235006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7344.239665] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7344.244060] Call Trace: [ 7344.245792] [ 7344.248381] ? show_regs.cold.9+0x22/0x2f [ 7344.250753] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7344.252351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7344.260660] ? watchdog+0x40/0x40 [ 7344.266109] ? hrtimer_interrupt+0x128/0x2c0 [ 7344.272965] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7344.277462] ? apic_timer_interrupt+0xf/0x20 [ 7344.278588] [ 7344.279105] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7344.283644] do_raw_spin_lock+0x177/0x190 [ 7344.290348] _raw_spin_lock+0x21/0x30 [ 7344.294349] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7344.298809] ? do_raw_spin_unlock+0x75/0x190 [ 7344.302765] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7344.309383] ? do_wait_intr+0xf0/0xf0 [ 7344.314560] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7344.318555] mdc_close+0x53b/0xd70 [mdc] [ 7344.322539] lmv_close+0x18d/0x490 [lmv] [ 7344.328499] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7344.336863] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7344.338830] ll_md_close+0x274/0x800 [lustre] [ 7344.343460] ll_file_release+0x128/0xc50 [lustre] [ 7344.352519] __fput+0x10d/0x440 [ 7344.357680] ____fput+0x12/0x20 [ 7344.364572] task_work_run+0xc6/0x110 [ 7344.370635] exit_to_usermode_loop+0x1dd/0x1f0 [ 7344.376845] do_syscall_64+0x3d6/0x440 [ 7344.378532] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7344.381884] RIP: 0033:0x7f1a63983cd8 [ 7344.384256] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7344.393254] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7344.398324] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7344.404460] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7344.409911] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7344.418092] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7344.422265] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7356.061199] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7356.071942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7356.150636] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7356.159222] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7356.168635] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7356.175703] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7356.191494] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7356.196891] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7356.204115] RDX: 00000000000001c9 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7356.207921] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7356.211722] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7356.214766] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7356.226476] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7356.235006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7356.238653] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7356.243225] Call Trace: [ 7356.244486] [ 7356.246360] ? show_regs.cold.9+0x22/0x2f [ 7356.250902] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7356.253970] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7356.259325] ? watchdog+0x40/0x40 [ 7356.261676] ? hrtimer_interrupt+0x128/0x2c0 [ 7356.264160] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7356.269243] ? apic_timer_interrupt+0xf/0x20 [ 7356.275262] [ 7356.278181] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7356.282277] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7356.284804] do_raw_spin_lock+0x177/0x190 [ 7356.286795] _raw_spin_lock+0x21/0x30 [ 7356.288145] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7356.292517] obd_mod_free+0x46/0x140 [ptlrpc] [ 7356.298629] mdc_commit_open+0x91/0xb0 [mdc] [ 7356.300985] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7356.303197] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7356.306094] after_reply+0xe2c/0x1780 [ptlrpc] [ 7356.310901] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7356.313197] ? __next_timer_interrupt+0x160/0x160 [ 7356.314402] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7356.315826] ? do_wait_intr+0xf0/0xf0 [ 7356.317126] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7356.319198] kthread+0x1d1/0x200 [ 7356.320329] ? set_kthread_struct+0x70/0x70 [ 7356.321534] ret_from_fork+0x1f/0x30 [ 7360.053259] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7360.062205] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7360.073630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7360.073713] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7360.073721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7360.073724] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7360.073738] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7360.073742] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7360.073746] RAX: 00000000000063d5 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7360.073749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7360.073751] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7360.073754] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7360.073756] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7360.073759] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7360.073762] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7360.073764] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7360.073774] Call Trace: [ 7360.073779] [ 7360.073785] ? show_regs.cold.9+0x22/0x2f [ 7360.073795] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7360.073802] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7360.073807] ? watchdog+0x40/0x40 [ 7360.073812] ? hrtimer_interrupt+0x128/0x2c0 [ 7360.073816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7360.073824] ? apic_timer_interrupt+0xf/0x20 [ 7360.073843] [ 7360.073846] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7360.073853] do_raw_spin_lock+0x177/0x190 [ 7360.073858] _raw_spin_lock+0x21/0x30 [ 7360.073863] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7360.076057] ? do_raw_spin_unlock+0x75/0x190 [ 7360.076068] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7360.076304] ? do_wait_intr+0xf0/0xf0 [ 7360.076312] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7360.085110] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7360.085191] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7360.085198] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7360.085202] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7360.085216] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7360.085220] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7360.085225] RAX: 0000000000006b86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7360.085227] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7360.085229] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7360.085231] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7360.085234] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7360.085237] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7360.085240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7360.085242] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7360.085251] Call Trace: [ 7360.085256] [ 7360.085262] ? show_regs.cold.9+0x22/0x2f [ 7360.085273] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7360.085279] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7360.085284] ? watchdog+0x40/0x40 [ 7360.085288] ? hrtimer_interrupt+0x128/0x2c0 [ 7360.085292] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7360.085301] ? apic_timer_interrupt+0xf/0x20 [ 7360.085305] [ 7360.085317] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7360.085323] do_raw_spin_lock+0x177/0x190 [ 7360.085328] _raw_spin_lock+0x21/0x30 [ 7360.085333] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7360.125349] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7360.125654] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7360.125865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7360.126176] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7360.126214] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7360.126244] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7360.126271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7360.126460] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7360.126693] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7360.126719] lmv_intent_open+0x338/0x1150 [lmv] [ 7360.126744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7360.126829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7360.126921] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7360.126943] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7360.127025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7360.127106] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7360.127187] ? do_raw_spin_unlock+0x75/0x190 [ 7360.127197] ? _raw_spin_unlock+0x12/0x30 [ 7360.127204] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7360.127232] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7360.127251] ll_atomic_open+0x36a/0xd70 [lustre] [ 7360.127643] lookup_open+0x356/0xb40 [ 7360.127652] do_last+0x313/0xfc0 [ 7360.127656] ? nd_jump_root+0xe5/0x160 [ 7360.127661] ? path_init+0x437/0x520 [ 7360.127665] path_openat+0xf7/0x500 [ 7360.127669] ? do_raw_spin_unlock+0x75/0x190 [ 7360.127674] do_filp_open+0x99/0x140 [ 7360.127679] ? getname_flags+0x6e/0x330 [ 7360.127683] ? __check_object_size+0xff/0x256 [ 7360.127690] ? do_raw_spin_unlock+0x75/0x190 [ 7360.127693] ? _raw_spin_unlock+0x12/0x30 [ 7360.127697] do_sys_openat2+0x2b4/0x410 [ 7360.127701] do_sys_open+0x73/0xa0 [ 7360.127705] __x64_sys_openat+0x24/0x30 [ 7360.127708] do_syscall_64+0xc1/0x440 [ 7360.127713] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7360.127718] RIP: 0033:0x7f1a63983332 [ 7360.127725] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7360.127728] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7360.127732] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7360.127735] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7360.127737] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7360.127739] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7360.127741] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7360.487635] ? do_raw_spin_unlock+0x75/0x190 [ 7360.491882] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7360.496804] ? do_wait_intr+0xf0/0xf0 [ 7360.499105] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7360.503320] mdc_close+0x53b/0xd70 [mdc] [ 7360.505312] lmv_close+0x18d/0x490 [lmv] [ 7360.507514] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7360.510736] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7360.513487] ll_md_close+0x274/0x800 [lustre] [ 7360.515888] ll_file_release+0x128/0xc50 [lustre] [ 7360.518943] __fput+0x10d/0x440 [ 7360.520617] ____fput+0x12/0x20 [ 7360.522797] task_work_run+0xc6/0x110 [ 7360.524622] exit_to_usermode_loop+0x1dd/0x1f0 [ 7360.527856] do_syscall_64+0x3d6/0x440 [ 7360.530266] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7360.533821] RIP: 0033:0x7f1a63983cd8 [ 7360.536115] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7360.552128] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7360.556730] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7360.560720] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7360.565063] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7360.569710] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7360.574084] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7366.113057] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5343s! [ 7366.117358] Showing busy workqueues and worker pools: [ 7366.123661] workqueue events: flags=0x0 [ 7366.128964] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7366.135205] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7366.152993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7366.161601] pending: sec_gc_main [ptlrpc] [ 7366.169315] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7366.182205] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7366.201064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7366.217729] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7366.234083] workqueue events_long: flags=0x0 [ 7366.243154] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7366.252190] pending: lu_objects_flush [obdclass] [ 7366.259689] workqueue events_unbound: flags=0x2 [ 7366.267621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7366.279749] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7366.298333] workqueue events_power_efficient: flags=0x80 [ 7366.307950] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7366.320030] pending: neigh_periodic_work, do_cache_clean [ 7366.330507] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7366.341113] pending: neigh_periodic_work, check_lifetime [ 7366.349085] workqueue mm_percpu_wq: flags=0x8 [ 7366.355743] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7366.373350] pending: vmstat_update [ 7366.383310] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7366.390766] pending: vmstat_update [ 7366.399159] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7366.413076] pending: vmstat_update [ 7366.419773] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7366.429805] pending: vmstat_update [ 7366.441453] workqueue writeback: flags=0x4a [ 7366.443843] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7366.456039] pending: wb_workfn, wb_workfn [ 7366.464882] workqueue xprtiod: flags=0xa [ 7366.474791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7366.485632] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7366.501541] workqueue ipv6_addrconf: flags=0x40008 [ 7366.503973] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7366.513225] pending: addrconf_verify_work [ 7366.520034] workqueue knbd0-recv: flags=0x1a [ 7366.528677] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7366.542907] in-flight: 647:recv_work [ 7366.550952] workqueue ptlrpc_pinger: flags=0x2 [ 7366.564015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7366.575688] pending: ptlrpc_pinger_main [ptlrpc] [ 7366.581334] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7356s workers=2 idle: 652 [ 7372.057496] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7372.067448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7372.125133] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7372.136590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7372.146134] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7372.155618] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7372.174690] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7372.181810] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7372.185739] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7372.191614] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7372.195836] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7372.199373] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7372.203132] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7372.207068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7372.209563] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7372.213967] Call Trace: [ 7372.215152] [ 7372.216134] ? show_regs.cold.9+0x22/0x2f [ 7372.218382] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7372.220637] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7372.222667] ? watchdog+0x40/0x40 [ 7372.224097] ? hrtimer_interrupt+0x128/0x2c0 [ 7372.226528] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7372.228695] ? apic_timer_interrupt+0xf/0x20 [ 7372.230711] [ 7372.231637] ? native_safe_halt+0x2e/0x50 [ 7372.233580] kvm_wait+0xcd/0xe0 [ 7372.234805] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7372.237381] do_raw_spin_lock+0x177/0x190 [ 7372.238999] _raw_spin_lock+0x21/0x30 [ 7372.241147] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7372.244142] ? do_raw_spin_unlock+0x75/0x190 [ 7372.246221] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7372.250515] ? do_wait_intr+0xf0/0xf0 [ 7372.253710] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7372.262516] mdc_close+0x53b/0xd70 [mdc] [ 7372.266100] lmv_close+0x18d/0x490 [lmv] [ 7372.274184] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7372.283528] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7372.291804] ll_md_close+0x274/0x800 [lustre] [ 7372.300217] ll_file_release+0x128/0xc50 [lustre] [ 7372.308799] __fput+0x10d/0x440 [ 7372.316090] ____fput+0x12/0x20 [ 7372.317964] task_work_run+0xc6/0x110 [ 7372.322310] exit_to_usermode_loop+0x1dd/0x1f0 [ 7372.329381] do_syscall_64+0x3d6/0x440 [ 7372.337077] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7372.344165] RIP: 0033:0x7f1a63983cd8 [ 7372.350088] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7372.371460] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7372.381156] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7372.388543] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7372.395184] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7372.403597] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7372.412737] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7384.058038] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7384.061802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7384.137746] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7384.150669] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7384.161346] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7384.164820] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7384.206847] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7384.217932] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7384.227400] RDX: 0000000000001a41 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7384.237988] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7384.246736] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7384.254253] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7384.261931] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7384.275769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7384.286686] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7384.291425] Call Trace: [ 7384.293014] [ 7384.294094] ? show_regs.cold.9+0x22/0x2f [ 7384.299610] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7384.304771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7384.308181] ? watchdog+0x40/0x40 [ 7384.310371] ? hrtimer_interrupt+0x128/0x2c0 [ 7384.312776] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7384.315414] ? apic_timer_interrupt+0xf/0x20 [ 7384.321854] [ 7384.324061] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7384.331779] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7384.337284] do_raw_spin_lock+0x177/0x190 [ 7384.340160] _raw_spin_lock+0x21/0x30 [ 7384.346855] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7384.354880] obd_mod_free+0x46/0x140 [ptlrpc] [ 7384.361907] mdc_commit_open+0x91/0xb0 [mdc] [ 7384.366809] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7384.369977] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7384.375962] after_reply+0xe2c/0x1780 [ptlrpc] [ 7384.382626] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7384.387802] ? __next_timer_interrupt+0x160/0x160 [ 7384.392266] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7384.397461] ? do_wait_intr+0xf0/0xf0 [ 7384.399375] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7384.405405] kthread+0x1d1/0x200 [ 7384.410099] ? set_kthread_struct+0x70/0x70 [ 7384.412089] ret_from_fork+0x1f/0x30 [ 7388.059672] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7388.069229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7388.131761] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7388.151390] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7388.171355] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7388.179225] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7388.199833] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7388.209851] RAX: 0000000000007293 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7388.222924] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7388.234004] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7388.248163] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7388.257981] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7388.263708] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7388.277925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7388.293763] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7388.305240] Call Trace: [ 7388.311096] [ 7388.312757] ? show_regs.cold.9+0x22/0x2f [ 7388.325961] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7388.333202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7388.336060] ? watchdog+0x40/0x40 [ 7388.345434] ? hrtimer_interrupt+0x128/0x2c0 [ 7388.350678] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7388.358326] ? apic_timer_interrupt+0xf/0x20 [ 7388.364595] [ 7388.366203] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7388.376015] do_raw_spin_lock+0x177/0x190 [ 7388.388334] _raw_spin_lock+0x21/0x30 [ 7388.390937] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7388.397879] ? do_raw_spin_unlock+0x75/0x190 [ 7388.406010] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7388.408789] ? do_wait_intr+0xf0/0xf0 [ 7388.416364] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7388.421622] mdc_close+0x53b/0xd70 [mdc] [ 7388.426968] lmv_close+0x18d/0x490 [lmv] [ 7388.435218] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7388.447061] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7388.452160] ll_md_close+0x274/0x800 [lustre] [ 7388.456855] ll_file_release+0x128/0xc50 [lustre] [ 7388.465454] __fput+0x10d/0x440 [ 7388.469515] ____fput+0x12/0x20 [ 7388.472778] task_work_run+0xc6/0x110 [ 7388.478060] exit_to_usermode_loop+0x1dd/0x1f0 [ 7388.484085] do_syscall_64+0x3d6/0x440 [ 7388.485425] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7388.492691] RIP: 0033:0x7f1a63983cd8 [ 7388.494865] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7388.519462] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7388.526763] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7388.535491] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7388.550147] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7388.558711] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7388.564989] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7392.057724] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7392.068637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7392.165054] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7392.190022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7392.217882] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7392.240639] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7392.284603] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7392.312260] RAX: 0000000000007912 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7392.335082] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7392.353197] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7392.375175] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7392.392828] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7392.405825] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7392.417065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7392.435890] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7392.439821] Call Trace: [ 7392.440764] [ 7392.441561] ? show_regs.cold.9+0x22/0x2f [ 7392.443374] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7392.445120] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7392.447382] ? watchdog+0x40/0x40 [ 7392.449011] ? hrtimer_interrupt+0x128/0x2c0 [ 7392.450911] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7392.454127] ? apic_timer_interrupt+0xf/0x20 [ 7392.456103] [ 7392.457484] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7392.460457] do_raw_spin_lock+0x177/0x190 [ 7392.462069] _raw_spin_lock+0x21/0x30 [ 7392.463960] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7392.466916] ? do_raw_spin_unlock+0x75/0x190 [ 7392.468602] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7392.470821] ? do_wait_intr+0xf0/0xf0 [ 7392.472620] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7392.475620] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7392.478115] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7392.481088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7392.484064] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7392.486501] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7392.488552] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7392.490760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7392.493100] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7392.496390] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7392.499362] lmv_intent_open+0x338/0x1150 [lmv] [ 7392.501802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7392.504895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7392.507998] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7392.510387] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7392.512624] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7392.515753] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7392.517925] ? do_raw_spin_unlock+0x75/0x190 [ 7392.520087] ? _raw_spin_unlock+0x12/0x30 [ 7392.522578] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7392.525638] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7392.527908] ll_atomic_open+0x36a/0xd70 [lustre] [ 7392.530738] lookup_open+0x356/0xb40 [ 7392.532408] do_last+0x313/0xfc0 [ 7392.534129] ? nd_jump_root+0xe5/0x160 [ 7392.535919] ? path_init+0x437/0x520 [ 7392.537587] path_openat+0xf7/0x500 [ 7392.539445] ? do_raw_spin_unlock+0x75/0x190 [ 7392.541114] do_filp_open+0x99/0x140 [ 7392.542700] ? getname_flags+0x6e/0x330 [ 7392.544714] ? __check_object_size+0xff/0x256 [ 7392.546930] ? do_raw_spin_unlock+0x75/0x190 [ 7392.549056] ? _raw_spin_unlock+0x12/0x30 [ 7392.551884] do_sys_openat2+0x2b4/0x410 [ 7392.554861] do_sys_open+0x73/0xa0 [ 7392.557024] __x64_sys_openat+0x24/0x30 [ 7392.558729] do_syscall_64+0xc1/0x440 [ 7392.560689] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7392.563790] RIP: 0033:0x7f1a63983332 [ 7392.565984] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7392.575085] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7392.579394] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7392.582953] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7392.586682] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7392.589786] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7392.597587] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7396.837068] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5373s! [ 7396.846370] Showing busy workqueues and worker pools: [ 7396.857532] workqueue events: flags=0x0 [ 7396.874046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7396.878434] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7396.892540] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7396.901786] pending: sec_gc_main [ptlrpc] [ 7396.915069] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7396.928168] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7396.950819] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7396.968421] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7396.986326] workqueue events_long: flags=0x0 [ 7396.990365] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7397.008983] pending: lu_objects_flush [obdclass] [ 7397.021850] workqueue events_unbound: flags=0x2 [ 7397.039021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7397.047293] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7397.063900] workqueue events_power_efficient: flags=0x80 [ 7397.079464] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7397.097151] pending: neigh_periodic_work, do_cache_clean [ 7397.109246] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7397.118095] pending: neigh_periodic_work, check_lifetime [ 7397.131071] workqueue mm_percpu_wq: flags=0x8 [ 7397.154021] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7397.156465] pending: vmstat_update [ 7397.179394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7397.199952] pending: vmstat_update [ 7397.206504] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7397.221051] pending: vmstat_update [ 7397.230490] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7397.243092] pending: vmstat_update [ 7397.257049] workqueue writeback: flags=0x4a [ 7397.264751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7397.275075] pending: wb_workfn, wb_workfn [ 7397.290141] workqueue xprtiod: flags=0xa [ 7397.300788] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7397.305944] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7397.315977] workqueue ipv6_addrconf: flags=0x40008 [ 7397.318987] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7397.329728] pending: addrconf_verify_work [ 7397.336136] workqueue knbd0-recv: flags=0x1a [ 7397.343108] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7397.349029] in-flight: 647:recv_work [ 7397.352149] workqueue ptlrpc_pinger: flags=0x2 [ 7397.359123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7397.367862] pending: ptlrpc_pinger_main [ptlrpc] [ 7397.372328] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7387s workers=2 idle: 652 [ 7400.056299] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7400.070389] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7400.146672] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7400.178667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7400.193640] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7400.197647] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7400.227835] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7400.236895] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7400.252439] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7400.269005] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7400.278471] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7400.290352] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7400.307299] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7400.319601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7400.332067] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7400.354807] Call Trace: [ 7400.358073] [ 7400.360924] ? show_regs.cold.9+0x22/0x2f [ 7400.372466] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7400.385153] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7400.392580] ? watchdog+0x40/0x40 [ 7400.404660] ? hrtimer_interrupt+0x128/0x2c0 [ 7400.413418] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7400.420168] ? apic_timer_interrupt+0xf/0x20 [ 7400.426536] [ 7400.429452] ? native_safe_halt+0x2e/0x50 [ 7400.441169] kvm_wait+0xcd/0xe0 [ 7400.451026] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7400.462951] do_raw_spin_lock+0x177/0x190 [ 7400.467364] _raw_spin_lock+0x21/0x30 [ 7400.470878] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7400.476361] ? do_raw_spin_unlock+0x75/0x190 [ 7400.487014] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7400.495391] ? do_wait_intr+0xf0/0xf0 [ 7400.502074] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7400.504113] mdc_close+0x53b/0xd70 [mdc] [ 7400.510113] lmv_close+0x18d/0x490 [lmv] [ 7400.511545] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7400.513929] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7400.519083] ll_md_close+0x274/0x800 [lustre] [ 7400.527942] ll_file_release+0x128/0xc50 [lustre] [ 7400.531539] __fput+0x10d/0x440 [ 7400.538178] ____fput+0x12/0x20 [ 7400.542425] task_work_run+0xc6/0x110 [ 7400.547640] exit_to_usermode_loop+0x1dd/0x1f0 [ 7400.551633] do_syscall_64+0x3d6/0x440 [ 7400.561204] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7400.573336] RIP: 0033:0x7f1a63983cd8 [ 7400.574644] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7400.594006] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7400.611459] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7400.616039] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7400.628798] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7400.637736] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7400.645831] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7412.064846] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 7412.087874] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7412.127247] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7412.135927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7412.142362] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7412.145628] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7412.152707] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7412.158600] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7412.165139] RDX: 000000000000461c RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7412.169931] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7412.173190] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7412.177193] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7412.179963] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7412.183643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7412.186747] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7412.189494] Call Trace: [ 7412.190426] [ 7412.191261] ? show_regs.cold.9+0x22/0x2f [ 7412.192834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7412.194874] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7412.196800] ? watchdog+0x40/0x40 [ 7412.198174] ? hrtimer_interrupt+0x128/0x2c0 [ 7412.199866] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7412.201895] ? apic_timer_interrupt+0xf/0x20 [ 7412.203844] [ 7412.204803] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7412.207393] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7412.210200] do_raw_spin_lock+0x177/0x190 [ 7412.211924] _raw_spin_lock+0x21/0x30 [ 7412.213531] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7412.216098] obd_mod_free+0x46/0x140 [ptlrpc] [ 7412.218224] mdc_commit_open+0x91/0xb0 [mdc] [ 7412.220105] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7412.222380] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7412.225517] after_reply+0xe2c/0x1780 [ptlrpc] [ 7412.227962] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7412.230179] ? __next_timer_interrupt+0x160/0x160 [ 7412.232347] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7412.234603] ? do_wait_intr+0xf0/0xf0 [ 7412.236216] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7412.238463] kthread+0x1d1/0x200 [ 7412.239957] ? set_kthread_struct+0x70/0x70 [ 7412.241714] ret_from_fork+0x1f/0x30 [ 7416.065090] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7416.078021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7416.151752] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7416.162770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7416.171856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7416.179986] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7416.202360] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7416.215029] RAX: 0000000000006e53 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7416.220051] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7416.232188] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7416.241044] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7416.249647] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7416.253621] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7416.258114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7416.265557] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7416.271250] Call Trace: [ 7416.276844] [ 7416.278705] ? show_regs.cold.9+0x22/0x2f [ 7416.288285] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7416.294974] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7416.299768] ? watchdog+0x40/0x40 [ 7416.304197] ? hrtimer_interrupt+0x128/0x2c0 [ 7416.307701] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7416.310552] ? apic_timer_interrupt+0xf/0x20 [ 7416.313178] [ 7416.314271] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7416.318251] do_raw_spin_lock+0x177/0x190 [ 7416.320611] _raw_spin_lock+0x21/0x30 [ 7416.322216] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7416.326015] ? do_raw_spin_unlock+0x75/0x190 [ 7416.328588] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7416.331336] ? do_wait_intr+0xf0/0xf0 [ 7416.333466] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7416.336273] mdc_close+0x53b/0xd70 [mdc] [ 7416.338572] lmv_close+0x18d/0x490 [lmv] [ 7416.341085] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7416.343344] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7416.345737] ll_md_close+0x274/0x800 [lustre] [ 7416.350605] ll_file_release+0x128/0xc50 [lustre] [ 7416.355306] __fput+0x10d/0x440 [ 7416.360601] ____fput+0x12/0x20 [ 7416.363076] task_work_run+0xc6/0x110 [ 7416.368114] exit_to_usermode_loop+0x1dd/0x1f0 [ 7416.375768] do_syscall_64+0x3d6/0x440 [ 7416.379891] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7416.383627] RIP: 0033:0x7f1a63983cd8 [ 7416.396625] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7416.415576] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7416.422930] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7416.431476] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7416.435581] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7416.440567] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7416.451415] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7424.053012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 7424.057997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7424.080703] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7424.085066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7424.089361] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7424.092530] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7424.103094] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7424.107922] RAX: 0000000000003d08 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7424.110745] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7424.114036] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7424.118378] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7424.121439] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7424.124639] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7424.129214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7424.131717] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7424.135409] Call Trace: [ 7424.136661] [ 7424.138116] ? show_regs.cold.9+0x22/0x2f [ 7424.140226] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7424.142685] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7424.146233] ? watchdog+0x40/0x40 [ 7424.148294] ? hrtimer_interrupt+0x128/0x2c0 [ 7424.151490] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7424.154531] ? apic_timer_interrupt+0xf/0x20 [ 7424.159159] [ 7424.160416] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7424.166062] do_raw_spin_lock+0x177/0x190 [ 7424.169302] _raw_spin_lock+0x21/0x30 [ 7424.171135] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7424.174365] ? do_raw_spin_unlock+0x75/0x190 [ 7424.177141] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7424.180089] ? do_wait_intr+0xf0/0xf0 [ 7424.182093] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7424.184989] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7424.187565] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7424.191118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7424.194167] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7424.197525] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7424.199806] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7424.202072] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7424.204512] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7424.207871] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7424.210692] lmv_intent_open+0x338/0x1150 [lmv] [ 7424.212685] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7424.215215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7424.218160] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7424.220591] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7424.222981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7424.225708] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7424.228076] ? do_raw_spin_unlock+0x75/0x190 [ 7424.230341] ? _raw_spin_unlock+0x12/0x30 [ 7424.232408] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7424.235094] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7424.238039] ll_atomic_open+0x36a/0xd70 [lustre] [ 7424.240430] lookup_open+0x356/0xb40 [ 7424.242229] do_last+0x313/0xfc0 [ 7424.244060] ? nd_jump_root+0xe5/0x160 [ 7424.245762] ? path_init+0x437/0x520 [ 7424.248204] path_openat+0xf7/0x500 [ 7424.250054] ? do_raw_spin_unlock+0x75/0x190 [ 7424.251802] do_filp_open+0x99/0x140 [ 7424.253684] ? getname_flags+0x6e/0x330 [ 7424.256526] ? __check_object_size+0xff/0x256 [ 7424.257994] ? do_raw_spin_unlock+0x75/0x190 [ 7424.260062] ? _raw_spin_unlock+0x12/0x30 [ 7424.261727] do_sys_openat2+0x2b4/0x410 [ 7424.263683] do_sys_open+0x73/0xa0 [ 7424.265233] __x64_sys_openat+0x24/0x30 [ 7424.266796] do_syscall_64+0xc1/0x440 [ 7424.268388] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7424.271373] RIP: 0033:0x7f1a63983332 [ 7424.272792] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7424.281513] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7424.284315] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7424.288169] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7424.291350] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7424.295450] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7424.299367] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7427.555740] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5404s! [ 7427.560868] Showing busy workqueues and worker pools: [ 7427.570191] workqueue events: flags=0x0 [ 7427.575133] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7427.578283] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7427.583568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7427.587040] pending: sec_gc_main [ptlrpc] [ 7427.595878] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7427.606627] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7427.625178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7427.633266] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7427.642772] workqueue events_long: flags=0x0 [ 7427.645531] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7427.652823] pending: lu_objects_flush [obdclass] [ 7427.659653] workqueue events_unbound: flags=0x2 [ 7427.664168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7427.667357] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7427.679599] workqueue events_power_efficient: flags=0x80 [ 7427.683144] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7427.688551] pending: neigh_periodic_work, do_cache_clean [ 7427.695527] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7427.700021] pending: neigh_periodic_work, check_lifetime [ 7427.702732] workqueue mm_percpu_wq: flags=0x8 [ 7427.707570] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7427.712232] pending: vmstat_update [ 7427.717273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7427.720980] pending: vmstat_update [ 7427.725770] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7427.728224] pending: vmstat_update [ 7427.734694] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7427.741869] pending: vmstat_update [ 7427.745141] workqueue writeback: flags=0x4a [ 7427.746620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7427.752678] pending: wb_workfn, wb_workfn [ 7427.756379] workqueue xprtiod: flags=0xa [ 7427.758231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7427.761175] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7427.778718] workqueue ipv6_addrconf: flags=0x40008 [ 7427.788262] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7427.798648] pending: addrconf_verify_work [ 7427.804184] workqueue knbd0-recv: flags=0x1a [ 7427.813529] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7427.815352] in-flight: 647:recv_work [ 7427.829016] workqueue ptlrpc_pinger: flags=0x2 [ 7427.840792] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7427.846049] pending: ptlrpc_pinger_main [ptlrpc] [ 7427.847759] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7417s workers=2 idle: 652 [ 7428.056869] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7428.060850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7428.107938] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7428.117219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7428.126620] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7428.132268] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7428.156229] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7428.161664] RAX: 0000000000002e01 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7428.174267] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7428.182048] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7428.189922] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7428.198614] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7428.205757] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7428.213753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7428.218284] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7428.227667] Call Trace: [ 7428.229434] [ 7428.230935] ? show_regs.cold.9+0x22/0x2f [ 7428.233800] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7428.235482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7428.242196] ? watchdog+0x40/0x40 [ 7428.245182] ? hrtimer_interrupt+0x128/0x2c0 [ 7428.253234] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7428.264673] ? apic_timer_interrupt+0xf/0x20 [ 7428.268312] [ 7428.270142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7428.273680] do_raw_spin_lock+0x177/0x190 [ 7428.277732] _raw_spin_lock+0x21/0x30 [ 7428.279359] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7428.283312] ? do_raw_spin_unlock+0x75/0x190 [ 7428.291667] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7428.296299] ? do_wait_intr+0xf0/0xf0 [ 7428.303451] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7428.305645] mdc_close+0x53b/0xd70 [mdc] [ 7428.316711] lmv_close+0x18d/0x490 [lmv] [ 7428.320561] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7428.329200] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7428.332046] ll_md_close+0x274/0x800 [lustre] [ 7428.339097] ll_file_release+0x128/0xc50 [lustre] [ 7428.346764] __fput+0x10d/0x440 [ 7428.352183] ____fput+0x12/0x20 [ 7428.358258] task_work_run+0xc6/0x110 [ 7428.359678] exit_to_usermode_loop+0x1dd/0x1f0 [ 7428.363565] do_syscall_64+0x3d6/0x440 [ 7428.369594] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7428.377783] RIP: 0033:0x7f1a63983cd8 [ 7428.379697] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7428.405695] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7428.414386] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7428.421397] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7428.430838] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7428.442709] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7428.459548] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7440.058489] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 7440.068808] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7440.117119] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7440.128214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7440.133152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7440.138565] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7440.160509] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7440.164815] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7440.169727] RDX: 00000000000031dc RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7440.177211] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7440.181768] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7440.186213] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7440.190778] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7440.201394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7440.208269] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7440.213125] Call Trace: [ 7440.214519] [ 7440.216091] ? show_regs.cold.9+0x22/0x2f [ 7440.220988] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7440.227480] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7440.231026] ? watchdog+0x40/0x40 [ 7440.232384] ? hrtimer_interrupt+0x128/0x2c0 [ 7440.242718] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7440.245690] ? apic_timer_interrupt+0xf/0x20 [ 7440.248397] [ 7440.250346] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7440.254501] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7440.260932] do_raw_spin_lock+0x177/0x190 [ 7440.263178] _raw_spin_lock+0x21/0x30 [ 7440.270220] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7440.276130] obd_mod_free+0x46/0x140 [ptlrpc] [ 7440.281592] mdc_commit_open+0x91/0xb0 [mdc] [ 7440.284132] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7440.294369] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7440.299777] after_reply+0xe2c/0x1780 [ptlrpc] [ 7440.303801] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7440.308136] ? __next_timer_interrupt+0x160/0x160 [ 7440.315037] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7440.317828] ? do_wait_intr+0xf0/0xf0 [ 7440.327169] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7440.331129] kthread+0x1d1/0x200 [ 7440.333945] ? set_kthread_struct+0x70/0x70 [ 7440.339649] ret_from_fork+0x1f/0x30 [ 7444.058777] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7444.062180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7444.108551] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7444.121078] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7444.129360] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7444.135222] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7444.151205] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7444.157901] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7444.160648] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7444.170193] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7444.185136] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7444.201603] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7444.209974] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7444.220089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7444.222047] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7444.231449] Call Trace: [ 7444.232250] [ 7444.233573] ? show_regs.cold.9+0x22/0x2f [ 7444.239662] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7444.243286] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7444.248454] ? watchdog+0x40/0x40 [ 7444.254395] ? hrtimer_interrupt+0x128/0x2c0 [ 7444.259908] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7444.266663] ? apic_timer_interrupt+0xf/0x20 [ 7444.272387] [ 7444.273171] ? native_safe_halt+0x2e/0x50 [ 7444.285122] kvm_wait+0xcd/0xe0 [ 7444.291584] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7444.302803] do_raw_spin_lock+0x177/0x190 [ 7444.310524] _raw_spin_lock+0x21/0x30 [ 7444.315935] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7444.322837] ? do_raw_spin_unlock+0x75/0x190 [ 7444.324403] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7444.332637] ? do_wait_intr+0xf0/0xf0 [ 7444.333684] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7444.342315] mdc_close+0x53b/0xd70 [mdc] [ 7444.357203] lmv_close+0x18d/0x490 [lmv] [ 7444.360406] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7444.371600] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7444.381114] ll_md_close+0x274/0x800 [lustre] [ 7444.382685] ll_file_release+0x128/0xc50 [lustre] [ 7444.384007] __fput+0x10d/0x440 [ 7444.386535] ____fput+0x12/0x20 [ 7444.388041] task_work_run+0xc6/0x110 [ 7444.392350] exit_to_usermode_loop+0x1dd/0x1f0 [ 7444.394163] do_syscall_64+0x3d6/0x440 [ 7444.395692] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7444.397536] RIP: 0033:0x7f1a63983cd8 [ 7444.399049] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7444.410502] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7444.413200] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7444.418905] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7444.423428] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7444.428102] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7444.431404] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7452.053722] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 7452.059175] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7452.099199] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7452.112915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7452.116950] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7452.119051] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7452.135203] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7452.138463] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7452.146046] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7452.149071] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7452.157240] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7452.162109] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7452.167846] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7452.175950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7452.180128] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7452.185231] Call Trace: [ 7452.186134] [ 7452.187266] ? show_regs.cold.9+0x22/0x2f [ 7452.191114] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7452.196440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7452.199585] ? watchdog+0x40/0x40 [ 7452.204114] ? hrtimer_interrupt+0x128/0x2c0 [ 7452.211941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7452.213298] ? apic_timer_interrupt+0xf/0x20 [ 7452.215908] [ 7452.217029] ? native_safe_halt+0x2e/0x50 [ 7452.219702] kvm_wait+0xcd/0xe0 [ 7452.222965] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7452.227181] do_raw_spin_lock+0x177/0x190 [ 7452.230772] _raw_spin_lock+0x21/0x30 [ 7452.234010] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7452.240149] ? do_raw_spin_unlock+0x75/0x190 [ 7452.246496] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7452.255108] ? do_wait_intr+0xf0/0xf0 [ 7452.258395] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7452.266247] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7452.273259] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7452.278767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7452.284922] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7452.289283] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7452.293870] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7452.297396] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7452.301800] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7452.305738] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7452.311400] lmv_intent_open+0x338/0x1150 [lmv] [ 7452.314364] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7452.318234] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7452.322410] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7452.325485] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7452.328579] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7452.332047] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7452.334953] ? do_raw_spin_unlock+0x75/0x190 [ 7452.338155] ? _raw_spin_unlock+0x12/0x30 [ 7452.340978] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7452.343490] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7452.346580] ll_atomic_open+0x36a/0xd70 [lustre] [ 7452.350078] lookup_open+0x356/0xb40 [ 7452.351911] do_last+0x313/0xfc0 [ 7452.354206] ? nd_jump_root+0xe5/0x160 [ 7452.356232] ? path_init+0x437/0x520 [ 7452.358861] path_openat+0xf7/0x500 [ 7452.361209] ? do_raw_spin_unlock+0x75/0x190 [ 7452.363386] do_filp_open+0x99/0x140 [ 7452.365353] ? getname_flags+0x6e/0x330 [ 7452.366976] ? __check_object_size+0xff/0x256 [ 7452.369771] ? do_raw_spin_unlock+0x75/0x190 [ 7452.371407] ? _raw_spin_unlock+0x12/0x30 [ 7452.374059] do_sys_openat2+0x2b4/0x410 [ 7452.376497] do_sys_open+0x73/0xa0 [ 7452.378804] __x64_sys_openat+0x24/0x30 [ 7452.380859] do_syscall_64+0xc1/0x440 [ 7452.382495] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7452.384887] RIP: 0033:0x7f1a63983332 [ 7452.386789] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7452.395823] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7452.399062] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7452.402943] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7452.405615] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7452.408566] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7452.411658] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7456.056395] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7456.060041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7456.093952] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7456.106193] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7456.115149] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7456.125567] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7456.139488] RSP: 0018:ffffc38542073af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7456.145846] RAX: 0000000000004d89 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7456.151509] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7456.158513] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7456.164273] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7456.171513] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7456.178292] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7456.188230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7456.190732] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7456.196467] Call Trace: [ 7456.199295] [ 7456.202026] ? show_regs.cold.9+0x22/0x2f [ 7456.204220] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7456.207997] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7456.209834] ? watchdog+0x40/0x40 [ 7456.214106] ? hrtimer_interrupt+0x128/0x2c0 [ 7456.216664] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7456.220128] ? apic_timer_interrupt+0xf/0x20 [ 7456.222270] [ 7456.228075] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7456.229750] do_raw_spin_lock+0x177/0x190 [ 7456.233521] _raw_spin_lock+0x21/0x30 [ 7456.235496] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7456.241784] ? do_raw_spin_unlock+0x75/0x190 [ 7456.245085] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7456.247993] ? do_wait_intr+0xf0/0xf0 [ 7456.249495] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7456.252036] mdc_close+0x53b/0xd70 [mdc] [ 7456.255041] lmv_close+0x18d/0x490 [lmv] [ 7456.257412] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7456.259512] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7456.263117] ll_md_close+0x274/0x800 [lustre] [ 7456.272424] ll_file_release+0x128/0xc50 [lustre] [ 7456.275727] __fput+0x10d/0x440 [ 7456.279180] ____fput+0x12/0x20 [ 7456.280206] task_work_run+0xc6/0x110 [ 7456.283767] exit_to_usermode_loop+0x1dd/0x1f0 [ 7456.285680] do_syscall_64+0x3d6/0x440 [ 7456.289030] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7456.292070] RIP: 0033:0x7f1a63983cd8 [ 7456.294312] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7456.315081] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7456.322317] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7456.329011] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7456.335376] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7456.342903] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7456.347715] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7458.274035] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5435s! [ 7458.281292] Showing busy workqueues and worker pools: [ 7458.287861] workqueue events: flags=0x0 [ 7458.290300] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7458.293852] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7458.298022] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7458.301176] pending: sec_gc_main [ptlrpc] [ 7458.303387] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7458.306988] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7458.311155] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7458.314003] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7458.318152] workqueue events_long: flags=0x0 [ 7458.319873] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7458.328246] pending: lu_objects_flush [obdclass] [ 7458.331067] workqueue events_unbound: flags=0x2 [ 7458.337562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7458.342391] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7458.356469] workqueue events_power_efficient: flags=0x80 [ 7458.361240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7458.370788] pending: neigh_periodic_work, do_cache_clean [ 7458.378851] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7458.381533] pending: neigh_periodic_work, check_lifetime [ 7458.389465] workqueue mm_percpu_wq: flags=0x8 [ 7458.395567] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7458.404015] pending: vmstat_update [ 7458.410437] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7458.414506] pending: vmstat_update [ 7458.420785] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7458.427440] pending: vmstat_update [ 7458.433571] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7458.439996] pending: vmstat_update [ 7458.444721] workqueue writeback: flags=0x4a [ 7458.449285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7458.459176] pending: wb_workfn, wb_workfn [ 7458.463306] workqueue xprtiod: flags=0xa [ 7458.469405] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7458.476099] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7458.485592] workqueue ipv6_addrconf: flags=0x40008 [ 7458.490272] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7458.495218] pending: addrconf_verify_work [ 7458.499678] workqueue knbd0-recv: flags=0x1a [ 7458.502363] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7458.508718] in-flight: 647:recv_work [ 7458.510093] workqueue ptlrpc_pinger: flags=0x2 [ 7458.515111] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7458.527336] pending: ptlrpc_pinger_main [ptlrpc] [ 7458.534041] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7448s workers=2 idle: 652 [ 7468.069612] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 7468.090058] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7468.211099] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7468.238154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7468.248154] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7468.257895] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7468.292062] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7468.301033] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7468.309977] RDX: 00000000000035ee RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7468.319376] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7468.331945] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7468.350462] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7468.361551] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7468.377239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7468.383005] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7468.391260] Call Trace: [ 7468.393579] [ 7468.396236] ? show_regs.cold.9+0x22/0x2f [ 7468.402187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7468.409563] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7468.412525] ? watchdog+0x40/0x40 [ 7468.415950] ? hrtimer_interrupt+0x128/0x2c0 [ 7468.422872] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7468.424651] ? apic_timer_interrupt+0xf/0x20 [ 7468.429252] [ 7468.433009] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7468.439229] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7468.441142] do_raw_spin_lock+0x177/0x190 [ 7468.446063] _raw_spin_lock+0x21/0x30 [ 7468.449964] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7468.455001] obd_mod_free+0x46/0x140 [ptlrpc] [ 7468.459043] mdc_commit_open+0x91/0xb0 [mdc] [ 7468.460949] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7468.468099] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7468.476672] after_reply+0xe2c/0x1780 [ptlrpc] [ 7468.478580] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7468.485158] ? __next_timer_interrupt+0x160/0x160 [ 7468.492221] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7468.495907] ? do_wait_intr+0xf0/0xf0 [ 7468.499471] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7468.505167] kthread+0x1d1/0x200 [ 7468.511065] ? set_kthread_struct+0x70/0x70 [ 7468.513259] ret_from_fork+0x1f/0x30 [ 7472.058404] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7472.063293] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7472.118342] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7472.125941] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7472.137411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7472.144271] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7472.157045] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7472.163744] RAX: 0000000000003dba RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7472.171681] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7472.179071] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7472.185566] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7472.203096] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7472.228408] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7472.243435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7472.253354] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7472.260707] Call Trace: [ 7472.263306] [ 7472.264300] ? show_regs.cold.9+0x22/0x2f [ 7472.268815] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7472.274389] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7472.276856] ? watchdog+0x40/0x40 [ 7472.278152] ? hrtimer_interrupt+0x128/0x2c0 [ 7472.291691] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7472.295432] ? apic_timer_interrupt+0xf/0x20 [ 7472.301756] [ 7472.302853] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7472.307905] do_raw_spin_lock+0x177/0x190 [ 7472.314290] _raw_spin_lock+0x21/0x30 [ 7472.320737] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7472.323262] ? do_raw_spin_unlock+0x75/0x190 [ 7472.325201] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7472.327942] ? do_wait_intr+0xf0/0xf0 [ 7472.329295] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7472.331998] mdc_close+0x53b/0xd70 [mdc] [ 7472.333460] lmv_close+0x18d/0x490 [lmv] [ 7472.335839] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7472.338374] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7472.340325] ll_md_close+0x274/0x800 [lustre] [ 7472.342595] ll_file_release+0x128/0xc50 [lustre] [ 7472.344352] __fput+0x10d/0x440 [ 7472.345425] ____fput+0x12/0x20 [ 7472.347205] task_work_run+0xc6/0x110 [ 7472.351273] exit_to_usermode_loop+0x1dd/0x1f0 [ 7472.353121] do_syscall_64+0x3d6/0x440 [ 7472.355093] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7472.360394] RIP: 0033:0x7f1a63983cd8 [ 7472.361963] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7472.612923] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7472.625830] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7472.636775] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7472.650202] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7472.658550] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7472.670276] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7482.547867] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 7482.557583] rcu: 3-...!: (3408393 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 7482.588234] (detected by 1, t=5460141 jiffies, g=269633, q=2046) [ 7482.601952] Sending NMI from CPU 1 to CPUs 3: [ 7482.621429] NMI backtrace for cpu 3 [ 7482.621436] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7482.621438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7482.621440] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7482.621441] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7482.621444] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 7482.621446] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7482.621448] RDX: 0000000000007e32 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7482.621449] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7482.621450] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7482.621451] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7482.621453] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7482.621457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7482.621459] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7482.621460] Call Trace: [ 7482.621461] [ 7482.621462] ? show_regs.cold.9+0x22/0x2f [ 7482.621463] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7482.621465] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7482.621466] ? nmi_handle+0xbd/0x210 [ 7482.621467] ? default_do_nmi+0x4b/0x1b0 [ 7482.621468] ? do_nmi+0x280/0x380 [ 7482.621469] ? end_repeat_nmi+0x16/0x1f [ 7482.621470] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7482.621471] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7482.621472] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7482.621473] [ 7482.621474] do_raw_spin_lock+0x177/0x190 [ 7482.621475] _raw_spin_lock+0x21/0x30 [ 7482.621476] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7482.621477] obd_mod_free+0x46/0x140 [ptlrpc] [ 7482.621479] mdc_commit_open+0x91/0xb0 [mdc] [ 7482.621480] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7482.621481] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7482.621482] after_reply+0xe2c/0x1780 [ptlrpc] [ 7482.621483] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7482.621484] ? __next_timer_interrupt+0x160/0x160 [ 7482.621486] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7482.621487] ? do_wait_intr+0xf0/0xf0 [ 7482.621488] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7482.621489] kthread+0x1d1/0x200 [ 7482.621490] ? set_kthread_struct+0x70/0x70 [ 7482.621491] ret_from_fork+0x1f/0x30 [ 7482.631595] rcu: rcu_sched kthread starved for 5459787 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 7483.098418] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7483.113254] rcu: RCU grace-period kthread stack dump: [ 7483.119753] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7483.135007] Call Trace: [ 7483.142735] __schedule+0x351/0xcb0 [ 7483.146694] schedule+0xc0/0x180 [ 7483.149988] schedule_timeout+0xb4/0x190 [ 7483.170319] ? do_raw_spin_unlock+0x75/0x190 [ 7483.171879] ? __next_timer_interrupt+0x160/0x160 [ 7483.183576] ? prepare_to_swait_event+0xf3/0x140 [ 7483.189178] rcu_gp_kthread+0x8dd/0x15b0 [ 7483.195886] ? force_qs_rnp+0x2c0/0x2c0 [ 7483.204951] kthread+0x1d1/0x200 [ 7483.206108] ? set_kthread_struct+0x70/0x70 [ 7483.213250] ret_from_fork+0x1f/0x30 [ 7483.219329] rcu: Stack dump where RCU GP kthread last ran: [ 7483.234678] Sending NMI from CPU 1 to CPUs 2: [ 7483.254450] NMI backtrace for cpu 2 [ 7483.254458] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7483.254460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7483.254461] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7483.254463] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7483.254465] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 [ 7483.254467] RAX: 0000000000005289 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7483.254469] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7483.254470] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7483.254847] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7483.254850] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7483.254851] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7483.254853] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7483.254854] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7483.254856] Call Trace: [ 7483.254856] [ 7483.254858] ? show_regs.cold.9+0x22/0x2f [ 7483.254859] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7483.254860] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7483.254861] ? nmi_handle+0xbd/0x210 [ 7483.254862] ? default_do_nmi+0x4b/0x1b0 [ 7483.254863] ? do_nmi+0x280/0x380 [ 7483.254864] ? end_repeat_nmi+0x16/0x1f [ 7483.254866] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7483.254867] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7483.254868] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7483.254869] [ 7483.254871] do_raw_spin_lock+0x177/0x190 [ 7483.254871] _raw_spin_lock+0x21/0x30 [ 7483.254872] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7483.254873] ? do_raw_spin_unlock+0x75/0x190 [ 7483.254875] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7483.254876] ? do_wait_intr+0xf0/0xf0 [ 7483.254877] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7483.254878] mdc_close+0x53b/0xd70 [mdc] [ 7483.254879] lmv_close+0x18d/0x490 [lmv] [ 7483.254880] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7483.254881] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7483.254882] ll_md_close+0x274/0x800 [lustre] [ 7483.254883] ll_file_release+0x128/0xc50 [lustre] [ 7483.254884] __fput+0x10d/0x440 [ 7483.254885] ____fput+0x12/0x20 [ 7483.254886] task_work_run+0xc6/0x110 [ 7483.254887] exit_to_usermode_loop+0x1dd/0x1f0 [ 7483.254888] do_syscall_64+0x3d6/0x440 [ 7483.254889] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7483.254890] RIP: 0033:0x7f1a63983cd8 [ 7483.254892] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7483.254894] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7483.254896] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7483.254898] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7483.254899] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7483.254900] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7483.254901] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7484.056092] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 7484.061779] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7484.109180] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7484.117908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7484.130223] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7484.135865] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7484.171167] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7484.177957] RAX: 0000000000007216 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7484.181942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7484.187004] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7484.194826] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7484.200239] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7484.210160] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7484.226670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7484.233192] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7484.241621] Call Trace: [ 7484.242867] [ 7484.243964] ? show_regs.cold.9+0x22/0x2f [ 7484.250107] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7484.256657] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7484.258922] ? watchdog+0x40/0x40 [ 7484.262255] ? hrtimer_interrupt+0x128/0x2c0 [ 7484.267211] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7484.273052] ? apic_timer_interrupt+0xf/0x20 [ 7484.276798] [ 7484.279095] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7484.285900] do_raw_spin_lock+0x177/0x190 [ 7484.288055] _raw_spin_lock+0x21/0x30 [ 7484.290061] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7484.297763] ? do_raw_spin_unlock+0x75/0x190 [ 7484.301924] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7484.308037] ? do_wait_intr+0xf0/0xf0 [ 7484.315619] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7484.322693] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7484.325770] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7484.333147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.336707] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7484.340032] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7484.343816] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7484.347526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.349118] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7484.363537] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7484.370795] lmv_intent_open+0x338/0x1150 [lmv] [ 7484.383931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.391706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.398212] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7484.411488] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7484.416204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.434505] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7484.439177] ? do_raw_spin_unlock+0x75/0x190 [ 7484.451160] ? _raw_spin_unlock+0x12/0x30 [ 7484.460103] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7484.467590] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7484.475689] ll_atomic_open+0x36a/0xd70 [lustre] [ 7484.481993] lookup_open+0x356/0xb40 [ 7484.490437] do_last+0x313/0xfc0 [ 7484.496728] ? nd_jump_root+0xe5/0x160 [ 7484.498005] ? path_init+0x437/0x520 [ 7484.499076] path_openat+0xf7/0x500 [ 7484.521133] ? do_raw_spin_unlock+0x75/0x190 [ 7484.525613] do_filp_open+0x99/0x140 [ 7484.528150] ? getname_flags+0x6e/0x330 [ 7484.537114] ? __check_object_size+0xff/0x256 [ 7484.542863] ? do_raw_spin_unlock+0x75/0x190 [ 7484.554106] ? _raw_spin_unlock+0x12/0x30 [ 7484.560979] do_sys_openat2+0x2b4/0x410 [ 7484.567210] do_sys_open+0x73/0xa0 [ 7484.573081] __x64_sys_openat+0x24/0x30 [ 7484.577082] do_syscall_64+0xc1/0x440 [ 7484.583071] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7484.589517] RIP: 0033:0x7f1a63983332 [ 7484.594009] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7484.632953] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7484.635195] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7484.665374] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7484.668230] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7484.691069] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7484.718949] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7488.997484] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5465s! [ 7489.000096] Showing busy workqueues and worker pools: [ 7489.001902] workqueue events: flags=0x0 [ 7489.003259] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7489.006208] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7489.008625] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7489.011326] pending: sec_gc_main [ptlrpc] [ 7489.013191] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7489.015093] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7489.020264] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7489.025177] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7489.029674] workqueue events_long: flags=0x0 [ 7489.031540] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7489.033707] pending: lu_objects_flush [obdclass] [ 7489.037303] workqueue events_unbound: flags=0x2 [ 7489.039935] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7489.042888] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7489.050063] workqueue events_power_efficient: flags=0x80 [ 7489.068974] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7489.079887] pending: neigh_periodic_work, do_cache_clean [ 7489.091368] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7489.102975] pending: neigh_periodic_work, check_lifetime [ 7489.104728] workqueue mm_percpu_wq: flags=0x8 [ 7489.120058] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7489.128141] pending: vmstat_update [ 7489.136800] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7489.147440] pending: vmstat_update [ 7489.149977] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7489.155021] pending: vmstat_update [ 7489.157042] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7489.169127] pending: vmstat_update [ 7489.170383] workqueue writeback: flags=0x4a [ 7489.178399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7489.188400] pending: wb_workfn, wb_workfn [ 7489.196111] workqueue xprtiod: flags=0xa [ 7489.204744] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7489.213533] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7489.223735] workqueue ipv6_addrconf: flags=0x40008 [ 7489.225941] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7489.234366] pending: addrconf_verify_work [ 7489.240294] workqueue knbd0-recv: flags=0x1a [ 7489.242580] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7489.248307] in-flight: 647:recv_work [ 7489.255539] workqueue ptlrpc_pinger: flags=0x2 [ 7489.258406] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7489.267765] pending: ptlrpc_pinger_main [ptlrpc] [ 7489.272577] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7479s workers=2 idle: 652 [ 7508.057184] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7508.057982] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7508.060032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7508.060045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7508.060189] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7508.060196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7508.060198] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7508.060222] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7508.060226] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7508.060230] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7508.060232] RDX: 0000000000007ac5 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7508.060234] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7508.060236] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7508.060238] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7508.060242] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7508.060245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7508.060247] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7508.060255] Call Trace: [ 7508.060272] [ 7508.060278] ? show_regs.cold.9+0x22/0x2f [ 7508.060288] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7508.060294] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7508.060299] ? watchdog+0x40/0x40 [ 7508.060303] ? hrtimer_interrupt+0x128/0x2c0 [ 7508.060307] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7508.060317] ? apic_timer_interrupt+0xf/0x20 [ 7508.060322] [ 7508.060325] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7508.060330] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7508.060334] do_raw_spin_lock+0x177/0x190 [ 7508.060340] _raw_spin_lock+0x21/0x30 [ 7508.060346] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7508.060605] obd_mod_free+0x46/0x140 [ptlrpc] [ 7508.060980] mdc_commit_open+0x91/0xb0 [mdc] [ 7508.061016] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7508.062277] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7508.062491] after_reply+0xe2c/0x1780 [ptlrpc] [ 7508.062699] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7508.063175] ? __next_timer_interrupt+0x160/0x160 [ 7508.063186] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7508.063393] ? do_wait_intr+0xf0/0xf0 [ 7508.063402] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7508.063596] kthread+0x1d1/0x200 [ 7508.063602] ? set_kthread_struct+0x70/0x70 [ 7508.063606] ret_from_fork+0x1f/0x30 [ 7508.063723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7508.063791] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7508.063796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7508.063799] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7508.063809] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7508.063813] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7508.063818] RAX: 0000000000005384 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7508.063820] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7508.063822] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7508.063824] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7508.063827] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7508.063830] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7508.063833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7508.063835] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7508.063845] Call Trace: [ 7508.063849] [ 7508.063852] ? show_regs.cold.9+0x22/0x2f [ 7508.063861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7508.063866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7508.063870] ? watchdog+0x40/0x40 [ 7508.063874] ? hrtimer_interrupt+0x128/0x2c0 [ 7508.063878] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7508.063884] ? apic_timer_interrupt+0xf/0x20 [ 7508.063889] [ 7508.063891] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7508.063897] do_raw_spin_lock+0x177/0x190 [ 7508.063902] _raw_spin_lock+0x21/0x30 [ 7508.063907] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7508.064667] ? do_raw_spin_unlock+0x75/0x190 [ 7508.064674] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7508.064873] ? do_wait_intr+0xf0/0xf0 [ 7508.064881] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7508.065447] mdc_close+0x53b/0xd70 [mdc] [ 7508.065484] lmv_close+0x18d/0x490 [lmv] [ 7508.065516] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7508.065659] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7508.076004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7508.076082] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7508.076288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7508.076292] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7508.076306] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7508.076311] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7508.076398] RAX: 0000000000004386 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7508.076402] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7508.076404] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7508.076406] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7508.076408] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7508.076411] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7508.076414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7508.076416] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7508.076424] Call Trace: [ 7508.076429] [ 7508.076435] ? show_regs.cold.9+0x22/0x2f [ 7508.076445] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7508.076450] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7508.076454] ? watchdog+0x40/0x40 [ 7508.076459] ? hrtimer_interrupt+0x128/0x2c0 [ 7508.076463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7508.076471] ? apic_timer_interrupt+0xf/0x20 [ 7508.076475] [ 7508.076478] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7508.076484] do_raw_spin_lock+0x177/0x190 [ 7508.076488] _raw_spin_lock+0x21/0x30 [ 7508.076493] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7508.078004] ? do_raw_spin_unlock+0x75/0x190 [ 7508.078015] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7508.079026] ? do_wait_intr+0xf0/0xf0 [ 7508.079036] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7508.079266] mdc_close+0x53b/0xd70 [mdc] [ 7508.079305] lmv_close+0x18d/0x490 [lmv] [ 7508.079424] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7508.086925] ll_md_close+0x274/0x800 [lustre] [ 7508.087494] ll_file_release+0x128/0xc50 [lustre] [ 7508.087589] __fput+0x10d/0x440 [ 7508.087599] ____fput+0x12/0x20 [ 7508.087603] task_work_run+0xc6/0x110 [ 7508.087611] exit_to_usermode_loop+0x1dd/0x1f0 [ 7508.087618] do_syscall_64+0x3d6/0x440 [ 7508.087623] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7508.087631] RIP: 0033:0x7f1a63983cd8 [ 7508.088183] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7508.088190] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7508.088194] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7508.088197] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7508.088199] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7508.088201] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7508.088204] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7508.913804] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7508.917636] ll_md_close+0x274/0x800 [lustre] [ 7508.922144] ll_file_release+0x128/0xc50 [lustre] [ 7508.924805] __fput+0x10d/0x440 [ 7508.926918] ____fput+0x12/0x20 [ 7508.928184] task_work_run+0xc6/0x110 [ 7508.931056] exit_to_usermode_loop+0x1dd/0x1f0 [ 7508.933772] do_syscall_64+0x3d6/0x440 [ 7508.935069] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7508.939941] RIP: 0033:0x7f1a63983cd8 [ 7508.943900] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7508.955869] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7508.963002] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7508.967046] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7508.973804] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7508.977511] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7508.983934] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7516.053709] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7516.056924] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7516.108693] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7516.116196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7516.121165] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7516.123471] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7516.131533] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7516.136601] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7516.142214] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7516.144455] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7516.149635] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7516.154079] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7516.157289] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7516.161944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7516.164207] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7516.167315] Call Trace: [ 7516.168337] [ 7516.169166] ? show_regs.cold.9+0x22/0x2f [ 7516.171071] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7516.173133] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7516.175209] ? watchdog+0x40/0x40 [ 7516.176771] ? hrtimer_interrupt+0x128/0x2c0 [ 7516.178473] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7516.180489] ? apic_timer_interrupt+0xf/0x20 [ 7516.184358] [ 7516.185598] ? native_safe_halt+0x2e/0x50 [ 7516.187573] kvm_wait+0xcd/0xe0 [ 7516.189167] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7516.191373] do_raw_spin_lock+0x177/0x190 [ 7516.193456] _raw_spin_lock+0x21/0x30 [ 7516.195227] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7516.197826] ? do_raw_spin_unlock+0x75/0x190 [ 7516.200591] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7516.203807] ? do_wait_intr+0xf0/0xf0 [ 7516.205267] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7516.206771] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7516.208307] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7516.213263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7516.215745] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7516.222436] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7516.228316] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7516.231642] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7516.233442] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7516.235651] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7516.237644] lmv_intent_open+0x338/0x1150 [lmv] [ 7516.239144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7516.240869] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7516.243000] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7516.246409] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7516.248315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7516.250361] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7516.252473] ? do_raw_spin_unlock+0x75/0x190 [ 7516.253610] ? _raw_spin_unlock+0x12/0x30 [ 7516.254689] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7516.256991] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7516.258373] ll_atomic_open+0x36a/0xd70 [lustre] [ 7516.260218] lookup_open+0x356/0xb40 [ 7516.261479] do_last+0x313/0xfc0 [ 7516.262418] ? nd_jump_root+0xe5/0x160 [ 7516.263417] ? path_init+0x437/0x520 [ 7516.264350] path_openat+0xf7/0x500 [ 7516.265356] ? do_raw_spin_unlock+0x75/0x190 [ 7516.266755] do_filp_open+0x99/0x140 [ 7516.267774] ? getname_flags+0x6e/0x330 [ 7516.268685] ? __check_object_size+0xff/0x256 [ 7516.270137] ? do_raw_spin_unlock+0x75/0x190 [ 7516.271189] ? _raw_spin_unlock+0x12/0x30 [ 7516.272268] do_sys_openat2+0x2b4/0x410 [ 7516.273298] do_sys_open+0x73/0xa0 [ 7516.274217] __x64_sys_openat+0x24/0x30 [ 7516.275210] do_syscall_64+0xc1/0x440 [ 7516.276370] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7516.277852] RIP: 0033:0x7f1a63983332 [ 7516.279133] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7516.286639] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7516.289698] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7516.292167] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7516.296456] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7516.300144] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7516.303412] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7519.714014] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5496s! [ 7519.723205] Showing busy workqueues and worker pools: [ 7519.742582] workqueue events: flags=0x0 [ 7519.746572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7519.758152] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7519.773581] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7519.782844] pending: sec_gc_main [ptlrpc] [ 7519.795920] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7519.807743] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7519.827969] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7519.841694] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7519.861697] workqueue events_long: flags=0x0 [ 7519.868096] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7519.881245] pending: lu_objects_flush [obdclass] [ 7519.890205] workqueue events_unbound: flags=0x2 [ 7519.899673] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7519.908693] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7519.922177] workqueue events_power_efficient: flags=0x80 [ 7519.930109] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7519.936173] pending: neigh_periodic_work, do_cache_clean [ 7519.940708] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7519.947395] pending: neigh_periodic_work, check_lifetime [ 7519.953662] workqueue mm_percpu_wq: flags=0x8 [ 7519.961761] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7519.968153] pending: vmstat_update [ 7519.970201] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7519.973971] pending: vmstat_update [ 7519.976517] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7519.980467] pending: vmstat_update [ 7519.982941] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7519.992507] pending: vmstat_update [ 7519.996567] workqueue writeback: flags=0x4a [ 7520.003956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7520.008948] pending: wb_workfn, wb_workfn [ 7520.011963] workqueue xprtiod: flags=0xa [ 7520.013243] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7520.019185] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7520.022841] workqueue ipv6_addrconf: flags=0x40008 [ 7520.025285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7520.032831] pending: addrconf_verify_work [ 7520.037009] workqueue knbd0-recv: flags=0x1a [ 7520.041594] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7520.047721] in-flight: 647:recv_work [ 7520.049440] workqueue ptlrpc_pinger: flags=0x2 [ 7520.054134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7520.059824] pending: ptlrpc_pinger_main [ptlrpc] [ 7520.063102] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7510s workers=2 idle: 652 [ 7536.057032] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7536.057689] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7536.058022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7536.058029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7536.058095] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7536.058100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7536.058102] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7536.058115] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7536.058120] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7536.058124] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7536.058126] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7536.058128] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7536.058129] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7536.058133] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7536.058136] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7536.058139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7536.058142] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7536.058152] Call Trace: [ 7536.058175] [ 7536.058181] ? show_regs.cold.9+0x22/0x2f [ 7536.058190] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7536.058196] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7536.058201] ? watchdog+0x40/0x40 [ 7536.058205] ? hrtimer_interrupt+0x128/0x2c0 [ 7536.058293] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7536.058301] ? apic_timer_interrupt+0xf/0x20 [ 7536.058306] [ 7536.058309] ? native_safe_halt+0x2e/0x50 [ 7536.058313] kvm_wait+0xcd/0xe0 [ 7536.058321] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7536.058328] do_raw_spin_lock+0x177/0x190 [ 7536.058333] _raw_spin_lock+0x21/0x30 [ 7536.058337] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7536.058635] obd_mod_free+0x46/0x140 [ptlrpc] [ 7536.058848] mdc_commit_open+0x91/0xb0 [mdc] [ 7536.058888] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7536.059099] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7536.059314] after_reply+0xe2c/0x1780 [ptlrpc] [ 7536.059524] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7536.059809] ? __next_timer_interrupt+0x160/0x160 [ 7536.059821] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7536.060075] ? do_wait_intr+0xf0/0xf0 [ 7536.060085] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7536.060341] kthread+0x1d1/0x200 [ 7536.060347] ? set_kthread_struct+0x70/0x70 [ 7536.060351] ret_from_fork+0x1f/0x30 [ 7536.066492] Modules linked in: [ 7536.076589] Modules linked in: [ 7536.084596] lustre(O) [ 7536.130545] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7536.130627] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7536.130634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7536.130637] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7536.130649] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7536.130653] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7536.130657] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7536.130660] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7536.130662] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7536.130664] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7536.130667] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7536.130671] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7536.130674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7536.130676] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7536.130684] Call Trace: [ 7536.130689] [ 7536.131370] ? show_regs.cold.9+0x22/0x2f [ 7536.149337] osp(O) [ 7536.151564] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7536.153241] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7536.153309] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7536.153316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7536.153319] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7536.153331] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7536.153335] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7536.153339] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7536.153341] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7536.153344] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7536.153346] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7536.153348] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7536.153351] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7536.153355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7536.153357] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7536.153371] Call Trace: [ 7536.153377] [ 7536.153381] ? show_regs.cold.9+0x22/0x2f [ 7536.153391] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7536.153397] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7536.153401] ? watchdog+0x40/0x40 [ 7536.153405] ? hrtimer_interrupt+0x128/0x2c0 [ 7536.153409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7536.153416] ? apic_timer_interrupt+0xf/0x20 [ 7536.153420] [ 7536.153429] ? native_safe_halt+0x2e/0x50 [ 7536.153434] kvm_wait+0xcd/0xe0 [ 7536.153441] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7536.153449] do_raw_spin_lock+0x177/0x190 [ 7536.153453] _raw_spin_lock+0x21/0x30 [ 7536.153457] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7536.153755] ? do_raw_spin_unlock+0x75/0x190 [ 7536.153761] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7536.154107] ? do_wait_intr+0xf0/0xf0 [ 7536.154115] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7536.154395] mdc_close+0x53b/0xd70 [mdc] [ 7536.154442] lmv_close+0x18d/0x490 [lmv] [ 7536.154478] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7536.154619] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7536.154770] ll_md_close+0x274/0x800 [lustre] [ 7536.154973] ll_file_release+0x128/0xc50 [lustre] [ 7536.155105] __fput+0x10d/0x440 [ 7536.155114] ____fput+0x12/0x20 [ 7536.155119] task_work_run+0xc6/0x110 [ 7536.155125] exit_to_usermode_loop+0x1dd/0x1f0 [ 7536.155131] do_syscall_64+0x3d6/0x440 [ 7536.155136] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7536.155142] RIP: 0033:0x7f1a63983cd8 [ 7536.155149] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7536.155153] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7536.155167] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7536.155170] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7536.155173] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7536.155175] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7536.155178] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7536.658920] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7536.669278] ? watchdog+0x40/0x40 [ 7536.672933] ? hrtimer_interrupt+0x128/0x2c0 [ 7536.679896] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7536.685416] ? apic_timer_interrupt+0xf/0x20 [ 7536.691363] [ 7536.693057] ? native_safe_halt+0x2e/0x50 [ 7536.696942] kvm_wait+0xcd/0xe0 [ 7536.699688] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7536.705956] do_raw_spin_lock+0x177/0x190 [ 7536.708053] _raw_spin_lock+0x21/0x30 [ 7536.712945] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7536.718647] ? do_raw_spin_unlock+0x75/0x190 [ 7536.721030] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7536.727681] ? do_wait_intr+0xf0/0xf0 [ 7536.733465] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7536.737773] mdc_close+0x53b/0xd70 [mdc] [ 7536.741337] lmv_close+0x18d/0x490 [lmv] [ 7536.743523] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7536.751123] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7536.759288] ll_md_close+0x274/0x800 [lustre] [ 7536.762080] ll_file_release+0x128/0xc50 [lustre] [ 7536.765498] __fput+0x10d/0x440 [ 7536.770001] ____fput+0x12/0x20 [ 7536.773961] task_work_run+0xc6/0x110 [ 7536.778476] exit_to_usermode_loop+0x1dd/0x1f0 [ 7536.782299] do_syscall_64+0x3d6/0x440 [ 7536.783564] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7536.788084] RIP: 0033:0x7f1a63983cd8 [ 7536.791273] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7536.804873] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7536.815181] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7536.819340] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7536.822540] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7536.825139] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7536.829478] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7548.053048] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7548.056133] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7548.106220] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7548.125562] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7548.134979] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7548.139175] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7548.161518] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7548.170038] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7548.189217] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7548.195371] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7548.214553] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7548.221091] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7548.228221] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7548.234453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7548.240923] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7548.251981] Call Trace: [ 7548.252915] [ 7548.261913] ? show_regs.cold.9+0x22/0x2f [ 7548.265997] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7548.272093] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7548.277356] ? watchdog+0x40/0x40 [ 7548.284928] ? hrtimer_interrupt+0x128/0x2c0 [ 7548.286472] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7548.297678] ? apic_timer_interrupt+0xf/0x20 [ 7548.299305] [ 7548.300136] ? native_safe_halt+0x2e/0x50 [ 7548.307264] kvm_wait+0xcd/0xe0 [ 7548.310458] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7548.313963] do_raw_spin_lock+0x177/0x190 [ 7548.319912] _raw_spin_lock+0x21/0x30 [ 7548.326785] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7548.331444] ? do_raw_spin_unlock+0x75/0x190 [ 7548.338212] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7548.341687] ? do_wait_intr+0xf0/0xf0 [ 7548.343253] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7548.353984] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7548.358497] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7548.362040] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7548.365591] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7548.373409] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7548.379455] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7548.388202] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7548.393326] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7548.402298] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7548.405874] lmv_intent_open+0x338/0x1150 [lmv] [ 7548.410278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7548.413569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7548.418358] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7548.421172] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7548.426363] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7548.429448] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7548.433809] ? do_raw_spin_unlock+0x75/0x190 [ 7548.442884] ? _raw_spin_unlock+0x12/0x30 [ 7548.449063] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7548.454229] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7548.457154] ll_atomic_open+0x36a/0xd70 [lustre] [ 7548.461946] lookup_open+0x356/0xb40 [ 7548.464550] do_last+0x313/0xfc0 [ 7548.467420] ? nd_jump_root+0xe5/0x160 [ 7548.476513] ? path_init+0x437/0x520 [ 7548.480354] path_openat+0xf7/0x500 [ 7548.482401] ? do_raw_spin_unlock+0x75/0x190 [ 7548.488687] do_filp_open+0x99/0x140 [ 7548.490030] ? getname_flags+0x6e/0x330 [ 7548.494542] ? __check_object_size+0xff/0x256 [ 7548.497217] ? do_raw_spin_unlock+0x75/0x190 [ 7548.498994] ? _raw_spin_unlock+0x12/0x30 [ 7548.500113] do_sys_openat2+0x2b4/0x410 [ 7548.501265] do_sys_open+0x73/0xa0 [ 7548.502528] __x64_sys_openat+0x24/0x30 [ 7548.504280] do_syscall_64+0xc1/0x440 [ 7548.506033] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7548.510643] RIP: 0033:0x7f1a63983332 [ 7548.511818] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7548.519186] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7548.522021] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7548.526467] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7548.530473] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7548.533253] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7548.535940] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7550.432792] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5527s! [ 7550.438057] Showing busy workqueues and worker pools: [ 7550.448040] workqueue events: flags=0x0 [ 7550.458948] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7550.479054] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7550.490010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7550.501015] pending: sec_gc_main [ptlrpc] [ 7550.509824] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7550.518130] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7550.533949] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7550.541234] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7550.564548] workqueue events_long: flags=0x0 [ 7550.574566] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7550.589841] pending: lu_objects_flush [obdclass] [ 7550.597864] workqueue events_unbound: flags=0x2 [ 7550.601193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7550.609551] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7550.616028] workqueue events_power_efficient: flags=0x80 [ 7550.626666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7550.631311] pending: neigh_periodic_work, do_cache_clean [ 7550.634620] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7550.648185] pending: neigh_periodic_work, check_lifetime [ 7550.655091] workqueue mm_percpu_wq: flags=0x8 [ 7550.657582] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7550.662119] pending: vmstat_update [ 7550.666109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7550.673381] pending: vmstat_update [ 7550.677633] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7550.685184] pending: vmstat_update [ 7550.690281] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7550.699201] pending: vmstat_update [ 7550.703695] workqueue writeback: flags=0x4a [ 7550.709354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7550.716816] pending: wb_workfn, wb_workfn [ 7550.718669] workqueue xprtiod: flags=0xa [ 7550.721656] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7550.724176] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7550.739305] workqueue ipv6_addrconf: flags=0x40008 [ 7550.745031] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7550.749122] pending: addrconf_verify_work [ 7550.752820] workqueue knbd0-recv: flags=0x1a [ 7550.754269] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7550.761472] in-flight: 647:recv_work [ 7550.765473] workqueue ptlrpc_pinger: flags=0x2 [ 7550.772492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7550.778118] pending: ptlrpc_pinger_main [ptlrpc] [ 7550.784638] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7540s workers=2 idle: 652 [ 7564.056025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7564.059384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7564.059546] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7564.060595] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7564.060603] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7564.060663] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7564.060670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7564.060673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7564.060687] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7564.060691] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7564.060696] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7564.060699] RDX: 000000000000291f RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7564.060701] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7564.060703] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7564.060706] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7564.060709] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7564.060712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7564.060715] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7564.060723] Call Trace: [ 7564.060729] [ 7564.060735] ? show_regs.cold.9+0x22/0x2f [ 7564.060745] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7564.060751] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7564.060756] ? watchdog+0x40/0x40 [ 7564.060760] ? hrtimer_interrupt+0x128/0x2c0 [ 7564.060764] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7564.060772] ? apic_timer_interrupt+0xf/0x20 [ 7564.060777] [ 7564.060780] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7564.060784] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7564.060789] do_raw_spin_lock+0x177/0x190 [ 7564.060794] _raw_spin_lock+0x21/0x30 [ 7564.060800] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7564.061063] obd_mod_free+0x46/0x140 [ptlrpc] [ 7564.061257] mdc_commit_open+0x91/0xb0 [mdc] [ 7564.061293] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7564.061487] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7564.061695] after_reply+0xe2c/0x1780 [ptlrpc] [ 7564.061889] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7564.062079] ? __next_timer_interrupt+0x160/0x160 [ 7564.062088] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7564.062289] ? do_wait_intr+0xf0/0xf0 [ 7564.062299] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7564.062499] kthread+0x1d1/0x200 [ 7564.062504] ? set_kthread_struct+0x70/0x70 [ 7564.062508] ret_from_fork+0x1f/0x30 [ 7564.088526] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7564.088540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7564.088543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7564.088625] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7564.088630] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7564.088635] RAX: 0000000000003b94 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7564.088638] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7564.088640] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7564.088643] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7564.088646] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7564.088649] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7564.088652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7564.088654] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7564.088663] Call Trace: [ 7564.088668] [ 7564.088674] ? show_regs.cold.9+0x22/0x2f [ 7564.088682] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7564.088689] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7564.088693] ? watchdog+0x40/0x40 [ 7564.088698] ? hrtimer_interrupt+0x128/0x2c0 [ 7564.088703] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7564.088712] ? apic_timer_interrupt+0xf/0x20 [ 7564.088717] [ 7564.088720] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7564.088727] do_raw_spin_lock+0x177/0x190 [ 7564.088732] _raw_spin_lock+0x21/0x30 [ 7564.088737] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7564.094213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7564.094297] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7564.094304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7564.094307] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7564.094320] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7564.094325] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7564.094329] RAX: 0000000000006b60 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7564.094331] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7564.094334] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7564.094336] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7564.094338] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7564.094342] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7564.094345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7564.094347] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7564.094355] Call Trace: [ 7564.094360] [ 7564.094365] ? show_regs.cold.9+0x22/0x2f [ 7564.094374] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7564.094381] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7564.094385] ? watchdog+0x40/0x40 [ 7564.094390] ? hrtimer_interrupt+0x128/0x2c0 [ 7564.094394] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7564.094402] ? apic_timer_interrupt+0xf/0x20 [ 7564.094406] [ 7564.094409] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7564.094415] do_raw_spin_lock+0x177/0x190 [ 7564.094420] _raw_spin_lock+0x21/0x30 [ 7564.094425] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7564.102330] ? do_raw_spin_unlock+0x75/0x190 [ 7564.102351] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7564.103212] ? do_wait_intr+0xf0/0xf0 [ 7564.103221] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7564.140981] ? do_raw_spin_unlock+0x75/0x190 [ 7564.141001] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7564.152196] mdc_close+0x53b/0xd70 [mdc] [ 7564.152247] lmv_close+0x18d/0x490 [lmv] [ 7564.152631] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7564.152763] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7564.152858] ll_md_close+0x274/0x800 [lustre] [ 7564.152948] ll_file_release+0x128/0xc50 [lustre] [ 7564.153052] __fput+0x10d/0x440 [ 7564.153061] ____fput+0x12/0x20 [ 7564.153066] task_work_run+0xc6/0x110 [ 7564.153074] exit_to_usermode_loop+0x1dd/0x1f0 [ 7564.153081] do_syscall_64+0x3d6/0x440 [ 7564.153086] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7564.153095] RIP: 0033:0x7f1a63983cd8 [ 7564.153103] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7564.153106] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7564.153112] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7564.153114] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7564.153117] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7564.153119] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7564.153122] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7564.739345] ? do_wait_intr+0xf0/0xf0 [ 7564.743304] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7564.752302] mdc_close+0x53b/0xd70 [mdc] [ 7564.760089] lmv_close+0x18d/0x490 [lmv] [ 7564.762444] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7564.770834] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7564.777897] ll_md_close+0x274/0x800 [lustre] [ 7564.782733] ll_file_release+0x128/0xc50 [lustre] [ 7564.787018] __fput+0x10d/0x440 [ 7564.790103] ____fput+0x12/0x20 [ 7564.793123] task_work_run+0xc6/0x110 [ 7564.794363] exit_to_usermode_loop+0x1dd/0x1f0 [ 7564.799638] do_syscall_64+0x3d6/0x440 [ 7564.801844] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7564.807918] RIP: 0033:0x7f1a63983cd8 [ 7564.816096] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7564.835993] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7564.843768] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7564.850366] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7564.857050] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7564.859460] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7564.866761] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7576.054395] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7576.063714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7576.130572] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7576.145077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7576.153127] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7576.157777] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7576.191993] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7576.209948] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7576.220737] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7576.239593] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7576.249913] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7576.264349] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7576.274880] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7576.280943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7576.287204] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7576.290729] Call Trace: [ 7576.291944] [ 7576.296695] ? show_regs.cold.9+0x22/0x2f [ 7576.298931] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7576.302857] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7576.308196] ? watchdog+0x40/0x40 [ 7576.312817] ? hrtimer_interrupt+0x128/0x2c0 [ 7576.318296] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7576.320053] ? apic_timer_interrupt+0xf/0x20 [ 7576.322722] [ 7576.323852] ? native_safe_halt+0x2e/0x50 [ 7576.329718] kvm_wait+0xcd/0xe0 [ 7576.333401] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7576.339001] do_raw_spin_lock+0x177/0x190 [ 7576.341149] _raw_spin_lock+0x21/0x30 [ 7576.344493] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7576.348757] ? do_raw_spin_unlock+0x75/0x190 [ 7576.350281] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7576.359139] ? do_wait_intr+0xf0/0xf0 [ 7576.365327] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7576.370296] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7576.373018] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7576.379873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7576.382020] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7576.387060] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7576.388781] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7576.402608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7576.409246] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7576.414820] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7576.424782] lmv_intent_open+0x338/0x1150 [lmv] [ 7576.428344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7576.437859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7576.442652] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7576.446014] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7576.453070] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7576.464867] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7576.472704] ? do_raw_spin_unlock+0x75/0x190 [ 7576.475123] ? _raw_spin_unlock+0x12/0x30 [ 7576.489378] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7576.491302] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7576.501310] ll_atomic_open+0x36a/0xd70 [lustre] [ 7576.511904] lookup_open+0x356/0xb40 [ 7576.517726] do_last+0x313/0xfc0 [ 7576.520291] ? nd_jump_root+0xe5/0x160 [ 7576.528330] ? path_init+0x437/0x520 [ 7576.532139] path_openat+0xf7/0x500 [ 7576.536589] ? do_raw_spin_unlock+0x75/0x190 [ 7576.541852] do_filp_open+0x99/0x140 [ 7576.545743] ? getname_flags+0x6e/0x330 [ 7576.547007] ? __check_object_size+0xff/0x256 [ 7576.549154] ? do_raw_spin_unlock+0x75/0x190 [ 7576.555181] ? _raw_spin_unlock+0x12/0x30 [ 7576.559165] do_sys_openat2+0x2b4/0x410 [ 7576.560665] do_sys_open+0x73/0xa0 [ 7576.561890] __x64_sys_openat+0x24/0x30 [ 7576.581724] do_syscall_64+0xc1/0x440 [ 7576.583020] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7576.588895] RIP: 0033:0x7f1a63983332 [ 7576.590200] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7576.613919] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7576.619024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7576.636930] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7576.647887] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7576.652988] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7576.655046] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7581.155154] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5558s! [ 7581.163307] Showing busy workqueues and worker pools: [ 7581.166992] workqueue events: flags=0x0 [ 7581.170760] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7581.174011] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7581.179718] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7581.182867] pending: sec_gc_main [ptlrpc] [ 7581.185387] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7581.188674] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7581.193048] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7581.196895] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7581.202797] workqueue events_long: flags=0x0 [ 7581.205134] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7581.211313] pending: lu_objects_flush [obdclass] [ 7581.215526] workqueue events_unbound: flags=0x2 [ 7581.218065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7581.223850] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7581.230272] workqueue events_power_efficient: flags=0x80 [ 7581.232991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7581.236685] pending: neigh_periodic_work, do_cache_clean [ 7581.239951] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7581.243526] pending: neigh_periodic_work, check_lifetime [ 7581.246013] workqueue mm_percpu_wq: flags=0x8 [ 7581.249947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7581.256903] pending: vmstat_update [ 7581.259031] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7581.267245] pending: vmstat_update [ 7581.272935] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7581.284722] pending: vmstat_update [ 7581.287488] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7581.294983] pending: vmstat_update [ 7581.298352] workqueue writeback: flags=0x4a [ 7581.302520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7581.307847] pending: wb_workfn, wb_workfn [ 7581.311514] workqueue xprtiod: flags=0xa [ 7581.313242] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7581.316273] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7581.319914] workqueue ipv6_addrconf: flags=0x40008 [ 7581.322846] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7581.328487] pending: addrconf_verify_work [ 7581.331054] workqueue knbd0-recv: flags=0x1a [ 7581.333785] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7581.337611] in-flight: 647:recv_work [ 7581.343063] workqueue ptlrpc_pinger: flags=0x2 [ 7581.345932] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7581.354670] pending: ptlrpc_pinger_main [ptlrpc] [ 7581.362326] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7571s workers=2 idle: 652 [ 7592.058100] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7592.058663] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7592.059420] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7592.059428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7592.059816] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7592.059822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7592.059825] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7592.059926] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7592.059932] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7592.059936] RAX: 0000000000003810 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7592.059939] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7592.059941] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7592.059943] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7592.059946] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7592.059949] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7592.059952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7592.059955] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7592.059963] Call Trace: [ 7592.059969] [ 7592.060027] ? show_regs.cold.9+0x22/0x2f [ 7592.060039] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7592.060046] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7592.060051] ? watchdog+0x40/0x40 [ 7592.060055] ? hrtimer_interrupt+0x128/0x2c0 [ 7592.060060] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7592.060069] ? apic_timer_interrupt+0xf/0x20 [ 7592.060074] [ 7592.060078] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7592.060084] do_raw_spin_lock+0x177/0x190 [ 7592.060089] _raw_spin_lock+0x21/0x30 [ 7592.060094] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7592.063413] ? do_raw_spin_unlock+0x75/0x190 [ 7592.063424] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7592.064360] ? do_wait_intr+0xf0/0xf0 [ 7592.064369] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7592.065700] mdc_close+0x53b/0xd70 [mdc] [ 7592.065738] lmv_close+0x18d/0x490 [lmv] [ 7592.065770] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7592.067208] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7592.070973] ll_md_close+0x274/0x800 [lustre] [ 7592.071366] ll_file_release+0x128/0xc50 [lustre] [ 7592.071452] __fput+0x10d/0x440 [ 7592.071462] ____fput+0x12/0x20 [ 7592.071466] task_work_run+0xc6/0x110 [ 7592.071559] exit_to_usermode_loop+0x1dd/0x1f0 [ 7592.071567] do_syscall_64+0x3d6/0x440 [ 7592.071571] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7592.071580] RIP: 0033:0x7f1a63983cd8 [ 7592.071587] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7592.071592] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7592.071597] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7592.071600] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7592.071602] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7592.071604] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7592.071606] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7592.074944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7592.075029] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7592.075047] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7592.075050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7592.075063] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7592.075067] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7592.075072] RAX: 00000000000079ad RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7592.075074] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7592.075077] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7592.075079] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7592.075082] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7592.075085] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7592.075089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7592.075091] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7592.075098] Call Trace: [ 7592.075103] [ 7592.075108] ? show_regs.cold.9+0x22/0x2f [ 7592.075124] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7592.075130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7592.075134] ? watchdog+0x40/0x40 [ 7592.075138] ? hrtimer_interrupt+0x128/0x2c0 [ 7592.075143] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7592.075155] ? apic_timer_interrupt+0xf/0x20 [ 7592.075159] [ 7592.075162] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7592.075168] do_raw_spin_lock+0x177/0x190 [ 7592.075173] _raw_spin_lock+0x21/0x30 [ 7592.075177] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7592.075687] ? do_raw_spin_unlock+0x75/0x190 [ 7592.075693] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7592.076138] ? do_wait_intr+0xf0/0xf0 [ 7592.076147] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7592.086917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7592.086999] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7592.087008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7592.087010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7592.087026] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7592.087031] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7592.087035] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7592.087038] RDX: 0000000000001f02 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7592.087040] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7592.087043] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7592.087045] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7592.087048] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7592.087051] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7592.087054] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7592.087063] Call Trace: [ 7592.087069] [ 7592.087074] ? show_regs.cold.9+0x22/0x2f [ 7592.087083] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7592.087089] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7592.087093] ? watchdog+0x40/0x40 [ 7592.087097] ? hrtimer_interrupt+0x128/0x2c0 [ 7592.087101] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7592.087109] ? apic_timer_interrupt+0xf/0x20 [ 7592.087113] [ 7592.087116] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7592.087122] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7592.087126] do_raw_spin_lock+0x177/0x190 [ 7592.087131] _raw_spin_lock+0x21/0x30 [ 7592.087136] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7592.088619] obd_mod_free+0x46/0x140 [ptlrpc] [ 7592.088822] mdc_commit_open+0x91/0xb0 [mdc] [ 7592.088861] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7592.089054] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7592.089735] after_reply+0xe2c/0x1780 [ptlrpc] [ 7592.089924] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7592.090107] ? __next_timer_interrupt+0x160/0x160 [ 7592.090117] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7592.096012] mdc_close+0x53b/0xd70 [mdc] [ 7592.096064] lmv_close+0x18d/0x490 [lmv] [ 7592.096098] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7592.096477] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7592.096658] ll_md_close+0x274/0x800 [lustre] [ 7592.096778] ll_file_release+0x128/0xc50 [lustre] [ 7592.096866] __fput+0x10d/0x440 [ 7592.096875] ____fput+0x12/0x20 [ 7592.096880] task_work_run+0xc6/0x110 [ 7592.096888] exit_to_usermode_loop+0x1dd/0x1f0 [ 7592.096896] do_syscall_64+0x3d6/0x440 [ 7592.096901] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7592.096948] RIP: 0033:0x7f1a63983cd8 [ 7592.096956] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7592.096960] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7592.096966] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7592.096968] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7592.096971] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7592.096973] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7592.096976] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7593.180847] ? do_wait_intr+0xf0/0xf0 [ 7593.185339] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7593.197350] kthread+0x1d1/0x200 [ 7593.198535] ? set_kthread_struct+0x70/0x70 [ 7593.200017] ret_from_fork+0x1f/0x30 [ 7608.053264] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7608.057949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7608.087099] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7608.094747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7608.099916] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.103039] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7608.114166] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7608.118363] RAX: 0000000000004734 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7608.122289] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7608.126239] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7608.130847] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7608.135610] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7608.139920] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7608.144999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7608.148103] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7608.152110] Call Trace: [ 7608.153617] [ 7608.154808] ? show_regs.cold.9+0x22/0x2f [ 7608.157090] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7608.161622] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7608.165224] ? watchdog+0x40/0x40 [ 7608.167744] ? hrtimer_interrupt+0x128/0x2c0 [ 7608.170252] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7608.174703] ? apic_timer_interrupt+0xf/0x20 [ 7608.177026] [ 7608.178297] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.181075] do_raw_spin_lock+0x177/0x190 [ 7608.182357] _raw_spin_lock+0x21/0x30 [ 7608.184419] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7608.187228] ? do_raw_spin_unlock+0x75/0x190 [ 7608.189804] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7608.193239] ? do_wait_intr+0xf0/0xf0 [ 7608.195294] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7608.198728] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7608.201817] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7608.205035] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7608.208613] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7608.212070] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7608.215302] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7608.218610] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7608.222073] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7608.225228] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7608.228867] lmv_intent_open+0x338/0x1150 [lmv] [ 7608.231681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7608.234977] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7608.239282] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7608.241494] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7608.244102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7608.247799] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7608.251064] ? do_raw_spin_unlock+0x75/0x190 [ 7608.253802] ? _raw_spin_unlock+0x12/0x30 [ 7608.255110] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7608.258158] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7608.260795] ll_atomic_open+0x36a/0xd70 [lustre] [ 7608.263264] lookup_open+0x356/0xb40 [ 7608.265863] do_last+0x313/0xfc0 [ 7608.266997] ? nd_jump_root+0xe5/0x160 [ 7608.269041] ? path_init+0x437/0x520 [ 7608.271614] path_openat+0xf7/0x500 [ 7608.273893] ? do_raw_spin_unlock+0x75/0x190 [ 7608.276220] do_filp_open+0x99/0x140 [ 7608.278687] ? getname_flags+0x6e/0x330 [ 7608.280804] ? __check_object_size+0xff/0x256 [ 7608.283745] ? do_raw_spin_unlock+0x75/0x190 [ 7608.286249] ? _raw_spin_unlock+0x12/0x30 [ 7608.288620] do_sys_openat2+0x2b4/0x410 [ 7608.290659] do_sys_open+0x73/0xa0 [ 7608.292481] __x64_sys_openat+0x24/0x30 [ 7608.294862] do_syscall_64+0xc1/0x440 [ 7608.296812] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7608.299743] RIP: 0033:0x7f1a63983332 [ 7608.301700] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7608.310643] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7608.313878] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7608.317653] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7608.323589] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7608.326926] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7608.330803] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7611.873109] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5588s! [ 7611.880714] Showing busy workqueues and worker pools: [ 7611.884199] workqueue events: flags=0x0 [ 7611.886986] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7611.891955] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7611.899229] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7611.904697] pending: sec_gc_main [ptlrpc] [ 7611.908710] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7611.913784] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7611.921435] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7611.925802] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7611.932714] workqueue events_long: flags=0x0 [ 7611.935881] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7611.940140] pending: lu_objects_flush [obdclass] [ 7611.944056] workqueue events_unbound: flags=0x2 [ 7611.947184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7611.951617] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7611.957609] workqueue events_power_efficient: flags=0x80 [ 7611.961900] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7611.967248] pending: neigh_periodic_work, do_cache_clean [ 7611.969924] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7611.974616] pending: neigh_periodic_work, check_lifetime [ 7611.978253] workqueue mm_percpu_wq: flags=0x8 [ 7611.980921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7611.985669] pending: vmstat_update [ 7611.989326] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7611.994032] pending: vmstat_update [ 7611.996241] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7612.000017] pending: vmstat_update [ 7612.003533] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7612.007739] pending: vmstat_update [ 7612.009725] workqueue writeback: flags=0x4a [ 7612.011872] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7612.014995] pending: wb_workfn, wb_workfn [ 7612.016878] workqueue xprtiod: flags=0xa [ 7612.019830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7612.023809] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7612.028267] workqueue ipv6_addrconf: flags=0x40008 [ 7612.031244] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7612.035032] pending: addrconf_verify_work [ 7612.038253] workqueue knbd0-recv: flags=0x1a [ 7612.040800] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7612.044226] in-flight: 647:recv_work [ 7612.046827] workqueue ptlrpc_pinger: flags=0x2 [ 7612.049414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7612.052835] pending: ptlrpc_pinger_main [ptlrpc] [ 7612.056701] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7602s workers=2 idle: 652 [ 7620.056075] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7620.057008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7620.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7620.058032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7620.058102] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7620.058107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7620.058110] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7620.058123] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7620.058127] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7620.058130] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7620.058133] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7620.058135] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7620.058138] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7620.058140] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7620.058143] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7620.058146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7620.058148] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7620.058155] Call Trace: [ 7620.058160] [ 7620.058165] ? show_regs.cold.9+0x22/0x2f [ 7620.058177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7620.058184] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7620.058188] ? watchdog+0x40/0x40 [ 7620.058193] ? hrtimer_interrupt+0x128/0x2c0 [ 7620.058197] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7620.058204] ? apic_timer_interrupt+0xf/0x20 [ 7620.058209] [ 7620.058211] ? native_safe_halt+0x2e/0x50 [ 7620.058215] kvm_wait+0xcd/0xe0 [ 7620.058223] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7620.058230] do_raw_spin_lock+0x177/0x190 [ 7620.058234] _raw_spin_lock+0x21/0x30 [ 7620.058239] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7620.058784] obd_mod_free+0x46/0x140 [ptlrpc] [ 7620.059086] mdc_commit_open+0x91/0xb0 [mdc] [ 7620.059122] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7620.059343] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7620.059672] after_reply+0xe2c/0x1780 [ptlrpc] [ 7620.059872] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7620.060172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata [ 7620.060244] ? __next_timer_interrupt+0x160/0x160 [ 7620.060255] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7620.060971] ? do_wait_intr+0xf0/0xf0 [ 7620.060980] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7620.061331] kthread+0x1d1/0x200 [ 7620.061337] ? set_kthread_struct+0x70/0x70 [ 7620.061341] ret_from_fork+0x1f/0x30 [ 7620.063126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7620.063201] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7620.063208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7620.063211] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7620.063224] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7620.063229] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7620.063233] RAX: 0000000000003eb0 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7620.063236] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7620.063239] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7620.063241] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7620.063244] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7620.063248] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7620.063251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7620.063253] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7620.063260] Call Trace: [ 7620.063265] [ 7620.063268] ? show_regs.cold.9+0x22/0x2f [ 7620.063277] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7620.063283] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7620.063288] ? watchdog+0x40/0x40 [ 7620.063292] ? hrtimer_interrupt+0x128/0x2c0 [ 7620.063296] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7620.063305] ? apic_timer_interrupt+0xf/0x20 [ 7620.063316] [ 7620.063319] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7620.063325] do_raw_spin_lock+0x177/0x190 [ 7620.063330] _raw_spin_lock+0x21/0x30 [ 7620.063334] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7620.063645] ? do_raw_spin_unlock+0x75/0x190 [ 7620.063650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7620.064099] ? do_wait_intr+0xf0/0xf0 [ 7620.064106] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7620.064296] mdc_close+0x53b/0xd70 [mdc] [ 7620.064335] lmv_close+0x18d/0x490 [lmv] [ 7620.064364] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7620.064519] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7620.064602] ll_md_close+0x274/0x800 [lustre] [ 7620.064686] ll_file_release+0x128/0xc50 [lustre] [ 7620.064764] __fput+0x10d/0x440 [ 7620.064772] ____fput+0x12/0x20 [ 7620.064775] task_work_run+0xc6/0x110 [ 7620.064781] exit_to_usermode_loop+0x1dd/0x1f0 [ 7620.064849] do_syscall_64+0x3d6/0x440 [ 7620.064854] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7620.064859] RIP: 0033:0x7f1a63983cd8 [ 7620.064865] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7620.064868] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7620.064873] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7620.064875] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7620.064939] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7620.064986] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7620.064989] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7620.614468] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7620.623855] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7620.633546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7620.641055] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7620.647711] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7620.660484] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7620.665418] RAX: 000000000000460e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7620.669632] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7620.675747] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7620.678806] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7620.683281] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7620.694752] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7620.705697] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7620.713062] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7620.721644] Call Trace: [ 7620.722789] [ 7620.724739] ? show_regs.cold.9+0x22/0x2f [ 7620.731631] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7620.737668] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7620.742523] ? watchdog+0x40/0x40 [ 7620.750885] ? hrtimer_interrupt+0x128/0x2c0 [ 7620.756532] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7620.761524] ? apic_timer_interrupt+0xf/0x20 [ 7620.764666] [ 7620.766233] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7620.771243] do_raw_spin_lock+0x177/0x190 [ 7620.775627] _raw_spin_lock+0x21/0x30 [ 7620.777774] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7620.781274] ? do_raw_spin_unlock+0x75/0x190 [ 7620.784265] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7620.794434] ? do_wait_intr+0xf0/0xf0 [ 7620.799430] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7620.803289] mdc_close+0x53b/0xd70 [mdc] [ 7620.805685] lmv_close+0x18d/0x490 [lmv] [ 7620.813300] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7620.818038] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7620.829571] ll_md_close+0x274/0x800 [lustre] [ 7620.832912] ll_file_release+0x128/0xc50 [lustre] [ 7620.836949] __fput+0x10d/0x440 [ 7620.839599] ____fput+0x12/0x20 [ 7620.840786] task_work_run+0xc6/0x110 [ 7620.845484] exit_to_usermode_loop+0x1dd/0x1f0 [ 7620.851472] do_syscall_64+0x3d6/0x440 [ 7620.854701] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7620.856689] RIP: 0033:0x7f1a63983cd8 [ 7620.857841] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7620.873850] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7620.877780] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7620.881211] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7620.884028] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7620.888152] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7620.892180] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7640.053025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7640.055803] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7640.094442] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7640.108069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7640.112013] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7640.113975] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7640.122988] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7640.125956] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7640.129315] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7640.133265] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7640.136780] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7640.140373] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7640.143916] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7640.147783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7640.150399] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7640.153620] Call Trace: [ 7640.154926] [ 7640.155934] ? show_regs.cold.9+0x22/0x2f [ 7640.157885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7640.160360] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7640.162486] ? watchdog+0x40/0x40 [ 7640.164264] ? hrtimer_interrupt+0x128/0x2c0 [ 7640.166317] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7640.168486] ? apic_timer_interrupt+0xf/0x20 [ 7640.170682] [ 7640.171608] ? native_safe_halt+0x2e/0x50 [ 7640.172963] kvm_wait+0xcd/0xe0 [ 7640.174023] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7640.176166] do_raw_spin_lock+0x177/0x190 [ 7640.177958] _raw_spin_lock+0x21/0x30 [ 7640.180709] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7640.185889] ? do_raw_spin_unlock+0x75/0x190 [ 7640.188897] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7640.194454] ? do_wait_intr+0xf0/0xf0 [ 7640.198972] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7640.201221] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7640.206573] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7640.213342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7640.219258] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7640.226735] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7640.228978] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7640.232387] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7640.237478] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7640.246159] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7640.252249] lmv_intent_open+0x338/0x1150 [lmv] [ 7640.255015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7640.258446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7640.265778] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7640.268683] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7640.271927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7640.277005] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7640.288381] ? do_raw_spin_unlock+0x75/0x190 [ 7640.293465] ? _raw_spin_unlock+0x12/0x30 [ 7640.298431] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7640.302242] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7640.306962] ll_atomic_open+0x36a/0xd70 [lustre] [ 7640.312307] lookup_open+0x356/0xb40 [ 7640.314447] do_last+0x313/0xfc0 [ 7640.317251] ? nd_jump_root+0xe5/0x160 [ 7640.320436] ? path_init+0x437/0x520 [ 7640.324495] path_openat+0xf7/0x500 [ 7640.327437] ? do_raw_spin_unlock+0x75/0x190 [ 7640.332463] do_filp_open+0x99/0x140 [ 7640.334985] ? getname_flags+0x6e/0x330 [ 7640.337275] ? __check_object_size+0xff/0x256 [ 7640.340142] ? do_raw_spin_unlock+0x75/0x190 [ 7640.342485] ? _raw_spin_unlock+0x12/0x30 [ 7640.345582] do_sys_openat2+0x2b4/0x410 [ 7640.347690] do_sys_open+0x73/0xa0 [ 7640.350212] __x64_sys_openat+0x24/0x30 [ 7640.353704] do_syscall_64+0xc1/0x440 [ 7640.357282] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7640.361009] RIP: 0033:0x7f1a63983332 [ 7640.362414] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7640.373484] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7640.376783] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7640.397736] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7640.404279] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7640.407339] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7640.416987] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7642.081046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5619s! [ 7642.093195] Showing busy workqueues and worker pools: [ 7642.101167] workqueue events: flags=0x0 [ 7642.108638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7642.121629] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7642.127711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7642.134543] pending: sec_gc_main [ptlrpc] [ 7642.140428] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7642.148014] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7642.158763] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7642.166391] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7642.172824] workqueue events_long: flags=0x0 [ 7642.176540] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7642.184489] pending: lu_objects_flush [obdclass] [ 7642.187581] workqueue events_unbound: flags=0x2 [ 7642.193393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7642.199558] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7642.212422] workqueue events_power_efficient: flags=0x80 [ 7642.214495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7642.220386] pending: neigh_periodic_work, do_cache_clean [ 7642.224466] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7642.229425] pending: neigh_periodic_work, check_lifetime [ 7642.235178] workqueue mm_percpu_wq: flags=0x8 [ 7642.236483] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7642.238470] pending: vmstat_update [ 7642.239686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7642.245382] pending: vmstat_update [ 7642.248316] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7642.250628] pending: vmstat_update [ 7642.254310] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7642.259309] pending: vmstat_update [ 7642.262966] workqueue writeback: flags=0x4a [ 7642.266309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7642.268288] pending: wb_workfn, wb_workfn [ 7642.269532] workqueue xprtiod: flags=0xa [ 7642.274290] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7642.278436] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7642.285022] workqueue ipv6_addrconf: flags=0x40008 [ 7642.288719] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7642.294146] pending: addrconf_verify_work [ 7642.296352] workqueue knbd0-recv: flags=0x1a [ 7642.299453] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7642.304460] in-flight: 647:recv_work [ 7642.307054] workqueue ptlrpc_pinger: flags=0x2 [ 7642.308396] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7642.310494] pending: ptlrpc_pinger_main [ptlrpc] [ 7642.312714] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7632s workers=2 idle: 652 [ 7648.056024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7648.057050] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7648.058044] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7648.058053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7648.058133] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7648.058136] Modules linked in: [ 7648.058141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7648.058143] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [ 7648.058148] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7648.058158] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 7648.058174] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7648.058177] ptlrpc(O) [ 7648.058180] obdclass(O) ksocklnd(O) [ 7648.058185] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 [ 7648.058188] lnet(O) libcfs(O) [ 7648.058192] ORIG_RAX: ffffffffffffff13 [ 7648.058195] rpcsec_gss_krb5 [ 7648.058198] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7648.058200] auth_rpcgss [ 7648.058203] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7648.058204] nfsv4 dns_resolver [ 7648.058210] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7648.058211] intel_rapl_msr intel_rapl_common [ 7648.058216] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7648.058220] sb_edac [ 7648.058223] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7648.058224] rapl pcspkr i2c_piix4 [ 7648.058230] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7648.058233] squashfs crct10dif_pclmul crc32_pclmul [ 7648.058239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7648.058242] crc32c_intel ata_generic [ 7648.058247] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7648.058249] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [ 7648.058260] Call Trace: [ 7648.058262] dm_mod sha512_ssse3 sha512_generic [ 7648.058268] [ 7648.058270] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7648.058277] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7648.058281] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7648.058278] ? show_regs.cold.9+0x22/0x2f [ 7648.058289] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7648.058296] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 [ 7648.058294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7648.058299] ORIG_RAX: ffffffffffffff13 [ 7648.058302] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7648.058306] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7648.058305] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7648.058310] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7648.058315] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7648.058315] ? watchdog+0x40/0x40 [ 7648.058320] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7648.058326] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7648.058326] ? hrtimer_interrupt+0x128/0x2c0 [ 7648.058331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7648.058336] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7648.058336] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7648.058343] Call Trace: [ 7648.058345] ? apic_timer_interrupt+0xf/0x20 [ 7648.058348] [ 7648.058354] [ 7648.058352] ? show_regs.cold.9+0x22/0x2f [ 7648.058359] ? native_safe_halt+0x2e/0x50 [ 7648.058363] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7648.058373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7648.058377] ? watchdog+0x40/0x40 [ 7648.058370] kvm_wait+0xcd/0xe0 [ 7648.058381] ? hrtimer_interrupt+0x128/0x2c0 [ 7648.058392] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7648.058399] ? apic_timer_interrupt+0xf/0x20 [ 7648.058403] [ 7648.058391] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7648.058405] ? native_safe_halt+0x2e/0x50 [ 7648.058417] kvm_wait+0xcd/0xe0 [ 7648.058419] do_raw_spin_lock+0x177/0x190 [ 7648.058425] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7648.058430] _raw_spin_lock+0x21/0x30 [ 7648.058434] do_raw_spin_lock+0x177/0x190 [ 7648.058443] _raw_spin_lock+0x21/0x30 [ 7648.058439] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7648.060540] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [ 7648.058448] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7648.060859] ? do_raw_spin_unlock+0x75/0x190 [ 7648.060866] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7648.061058] ? do_wait_intr+0xf0/0xf0 [ 7648.061064] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7648.061257] mdc_close+0x53b/0xd70 [mdc] [ 7648.061292] lmv_close+0x18d/0x490 [lmv] [ 7648.061314] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7648.061415] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7648.061491] ll_md_close+0x274/0x800 [lustre] [ 7648.061689] ll_file_release+0x128/0xc50 [lustre] [ 7648.061794] __fput+0x10d/0x440 [ 7648.061801] ____fput+0x12/0x20 [ 7648.061805] task_work_run+0xc6/0x110 [ 7648.061811] exit_to_usermode_loop+0x1dd/0x1f0 [ 7648.061817] do_syscall_64+0x3d6/0x440 [ 7648.061821] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7648.061828] RIP: 0033:0x7f1a63983cd8 [ 7648.061834] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7648.061838] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7648.061841] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7648.061844] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7648.061846] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7648.061848] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7648.061850] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7648.063257] obd_mod_free+0x46/0x140 [ptlrpc] [ 7648.063484] mdc_commit_open+0x91/0xb0 [mdc] [ 7648.063515] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7648.077411] sha512_generic [ 7648.077422] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7648.077428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7648.077430] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7648.077442] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7648.077445] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7648.077449] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7648.077451] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7648.077453] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7648.077455] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7648.077457] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7648.077460] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7648.077463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7648.077466] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7648.077473] Call Trace: [ 7648.077478] [ 7648.077483] ? show_regs.cold.9+0x22/0x2f [ 7648.077493] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7648.077500] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7648.077505] ? watchdog+0x40/0x40 [ 7648.077508] ? hrtimer_interrupt+0x128/0x2c0 [ 7648.077511] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7648.077517] ? apic_timer_interrupt+0xf/0x20 [ 7648.077521] [ 7648.077523] ? native_safe_halt+0x2e/0x50 [ 7648.077527] kvm_wait+0xcd/0xe0 [ 7648.077533] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7648.077540] do_raw_spin_lock+0x177/0x190 [ 7648.077544] _raw_spin_lock+0x21/0x30 [ 7648.077547] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7648.080775] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7648.081041] after_reply+0xe2c/0x1780 [ptlrpc] [ 7648.081241] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7648.081430] ? __next_timer_interrupt+0x160/0x160 [ 7648.081440] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7648.081635] ? do_wait_intr+0xf0/0xf0 [ 7648.081644] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7648.082006] kthread+0x1d1/0x200 [ 7648.082012] ? set_kthread_struct+0x70/0x70 [ 7648.082015] ret_from_fork+0x1f/0x30 [ 7648.358484] ? do_raw_spin_unlock+0x75/0x190 [ 7648.359679] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7648.361421] ? do_wait_intr+0xf0/0xf0 [ 7648.362478] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7648.364431] mdc_close+0x53b/0xd70 [mdc] [ 7648.365644] lmv_close+0x18d/0x490 [lmv] [ 7648.366836] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7648.368580] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7648.370196] ll_md_close+0x274/0x800 [lustre] [ 7648.371813] ll_file_release+0x128/0xc50 [lustre] [ 7648.373244] __fput+0x10d/0x440 [ 7648.374365] ____fput+0x12/0x20 [ 7648.375391] task_work_run+0xc6/0x110 [ 7648.377171] exit_to_usermode_loop+0x1dd/0x1f0 [ 7648.378507] do_syscall_64+0x3d6/0x440 [ 7648.379490] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7648.381150] RIP: 0033:0x7f1a63983cd8 [ 7648.382140] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7648.388026] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7648.390389] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7648.392401] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7648.394469] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7648.396926] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7648.399353] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7662.551122] rcu: INFO: rcu_sched self-detected stall on CPU [ 7662.559181] rcu: 3-...!: (3534711 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 7662.571225] (t=5640165 jiffies g=269633 q=2116) [ 7662.575549] rcu: rcu_sched kthread starved for 5639815 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 7662.587011] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7662.595366] rcu: RCU grace-period kthread stack dump: [ 7662.600217] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7662.606906] Call Trace: [ 7662.608454] __schedule+0x351/0xcb0 [ 7662.611320] schedule+0xc0/0x180 [ 7662.613454] schedule_timeout+0xb4/0x190 [ 7662.617179] ? do_raw_spin_unlock+0x75/0x190 [ 7662.620912] ? __next_timer_interrupt+0x160/0x160 [ 7662.626306] ? prepare_to_swait_event+0xf3/0x140 [ 7662.632105] rcu_gp_kthread+0x8dd/0x15b0 [ 7662.635351] ? force_qs_rnp+0x2c0/0x2c0 [ 7662.637460] kthread+0x1d1/0x200 [ 7662.644048] ? set_kthread_struct+0x70/0x70 [ 7662.651123] ret_from_fork+0x1f/0x30 [ 7662.654366] rcu: Stack dump where RCU GP kthread last ran: [ 7662.659431] Sending NMI from CPU 3 to CPUs 1: [ 7662.663155] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 7662.669629] NMI backtrace for cpu 3 [ 7662.671701] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7662.686427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7662.696888] Call Trace: [ 7662.698036] [ 7662.701003] ? dump_stack+0xbb/0x10e [ 7662.704373] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7662.707223] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7662.710442] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7662.714983] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7662.718440] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7662.721077] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7662.723739] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7662.726230] ? update_process_times+0x6d/0xc0 [ 7662.728203] ? tick_sched_handle.isra.18+0x27/0x90 [ 7662.730253] ? tick_sched_timer+0x53/0xb0 [ 7662.732238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7662.734339] ? tick_sched_handle.isra.18+0x90/0x90 [ 7662.736910] ? hrtimer_interrupt+0x128/0x2c0 [ 7662.739071] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7662.741116] ? apic_timer_interrupt+0xf/0x20 [ 7662.743436] [ 7662.744715] ? native_safe_halt+0x2e/0x50 [ 7662.746310] ? kvm_wait+0xcd/0xe0 [ 7662.747827] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7662.750732] ? do_raw_spin_lock+0x177/0x190 [ 7662.754999] ? _raw_spin_lock+0x21/0x30 [ 7662.758246] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7662.761197] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7662.765032] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7662.770793] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7662.772901] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7662.776323] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7662.783132] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7662.786374] ? __next_timer_interrupt+0x160/0x160 [ 7662.790150] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7662.793397] ? do_wait_intr+0xf0/0xf0 [ 7662.796565] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7662.799538] ? kthread+0x1d1/0x200 [ 7662.801299] ? set_kthread_struct+0x70/0x70 [ 7662.803303] ? ret_from_fork+0x1f/0x30 [ 7668.056242] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7668.060863] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7668.117346] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7668.129701] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7668.136373] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7668.151721] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7668.166121] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7668.169569] RAX: 0000000000001aa4 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7668.172844] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7668.176140] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7668.179014] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7668.181316] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7668.184069] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7668.187480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7668.190045] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7668.198117] Call Trace: [ 7668.199157] [ 7668.200097] ? show_regs.cold.9+0x22/0x2f [ 7668.205509] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7668.207519] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7668.210402] ? watchdog+0x40/0x40 [ 7668.211696] ? hrtimer_interrupt+0x128/0x2c0 [ 7668.213005] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7668.215914] ? apic_timer_interrupt+0xf/0x20 [ 7668.218868] [ 7668.219850] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7668.222751] do_raw_spin_lock+0x177/0x190 [ 7668.224890] _raw_spin_lock+0x21/0x30 [ 7668.227122] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7668.230380] ? do_raw_spin_unlock+0x75/0x190 [ 7668.233487] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7668.236375] ? do_wait_intr+0xf0/0xf0 [ 7668.238245] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7668.240890] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7668.243956] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7668.247828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7668.250422] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7668.252260] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7668.253984] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7668.256816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7668.259461] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7668.262713] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7668.265212] lmv_intent_open+0x338/0x1150 [lmv] [ 7668.267765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7668.270483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7668.273745] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7668.276261] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7668.278323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7668.281043] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7668.283527] ? do_raw_spin_unlock+0x75/0x190 [ 7668.285507] ? _raw_spin_unlock+0x12/0x30 [ 7668.287788] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7668.290263] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7668.292731] ll_atomic_open+0x36a/0xd70 [lustre] [ 7668.295538] lookup_open+0x356/0xb40 [ 7668.297458] do_last+0x313/0xfc0 [ 7668.299043] ? nd_jump_root+0xe5/0x160 [ 7668.300499] ? path_init+0x437/0x520 [ 7668.302392] path_openat+0xf7/0x500 [ 7668.304018] ? do_raw_spin_unlock+0x75/0x190 [ 7668.306067] do_filp_open+0x99/0x140 [ 7668.308329] ? getname_flags+0x6e/0x330 [ 7668.310043] ? __check_object_size+0xff/0x256 [ 7668.312313] ? do_raw_spin_unlock+0x75/0x190 [ 7668.314500] ? _raw_spin_unlock+0x12/0x30 [ 7668.317060] do_sys_openat2+0x2b4/0x410 [ 7668.318918] do_sys_open+0x73/0xa0 [ 7668.320615] __x64_sys_openat+0x24/0x30 [ 7668.322011] do_syscall_64+0xc1/0x440 [ 7668.324175] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7668.326966] RIP: 0033:0x7f1a63983332 [ 7668.328511] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7668.338029] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7668.341479] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7668.345060] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7668.349087] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7668.352798] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7668.356251] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7672.803463] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5649s! [ 7672.814909] Showing busy workqueues and worker pools: [ 7672.818303] workqueue events: flags=0x0 [ 7672.821837] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7672.826117] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7672.832270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7672.835815] pending: sec_gc_main [ptlrpc] [ 7672.838119] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7672.840775] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7672.846039] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7672.849368] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7672.853630] workqueue events_long: flags=0x0 [ 7672.855074] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7672.857691] pending: lu_objects_flush [obdclass] [ 7672.860301] workqueue events_unbound: flags=0x2 [ 7672.863936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7672.869017] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7672.880749] workqueue events_power_efficient: flags=0x80 [ 7672.883974] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7672.891455] pending: neigh_periodic_work, do_cache_clean [ 7672.894605] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7672.904089] pending: neigh_periodic_work, check_lifetime [ 7672.911119] workqueue mm_percpu_wq: flags=0x8 [ 7672.914883] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7672.918240] pending: vmstat_update [ 7672.920048] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7672.922941] pending: vmstat_update [ 7672.925167] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7672.928083] pending: vmstat_update [ 7672.929813] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7672.932780] pending: vmstat_update [ 7672.934313] workqueue writeback: flags=0x4a [ 7672.936364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7672.941652] pending: wb_workfn, wb_workfn [ 7672.945726] workqueue xprtiod: flags=0xa [ 7672.949308] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7672.954173] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7672.963965] workqueue ipv6_addrconf: flags=0x40008 [ 7672.966785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7672.969940] pending: addrconf_verify_work [ 7672.973902] workqueue knbd0-recv: flags=0x1a [ 7672.977753] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7672.988200] in-flight: 647:recv_work [ 7672.992133] workqueue ptlrpc_pinger: flags=0x2 [ 7673.002472] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7673.012182] pending: ptlrpc_pinger_main [ptlrpc] [ 7673.017723] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7663s workers=2 idle: 652 [ 7676.057096] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7676.060023] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7676.105629] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7676.118559] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7676.124817] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7676.130122] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7676.152103] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7676.157847] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7676.169707] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7676.181535] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7676.194799] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7676.197051] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7676.208553] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7676.216301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7676.220083] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7676.225593] Call Trace: [ 7676.226992] [ 7676.228029] ? show_regs.cold.9+0x22/0x2f [ 7676.230330] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7676.232656] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7676.235140] ? watchdog+0x40/0x40 [ 7676.237764] ? hrtimer_interrupt+0x128/0x2c0 [ 7676.240072] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7676.242428] ? apic_timer_interrupt+0xf/0x20 [ 7676.244547] [ 7676.245620] ? native_safe_halt+0x2e/0x50 [ 7676.247459] kvm_wait+0xcd/0xe0 [ 7676.249245] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7676.251753] do_raw_spin_lock+0x177/0x190 [ 7676.254031] _raw_spin_lock+0x21/0x30 [ 7676.255411] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7676.258118] ? do_raw_spin_unlock+0x75/0x190 [ 7676.260028] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7676.262318] ? do_wait_intr+0xf0/0xf0 [ 7676.264037] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7676.266613] mdc_close+0x53b/0xd70 [mdc] [ 7676.269110] lmv_close+0x18d/0x490 [lmv] [ 7676.271041] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7676.277254] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7676.279091] ll_md_close+0x274/0x800 [lustre] [ 7676.284292] ll_file_release+0x128/0xc50 [lustre] [ 7676.287265] __fput+0x10d/0x440 [ 7676.288689] ____fput+0x12/0x20 [ 7676.289761] task_work_run+0xc6/0x110 [ 7676.291058] exit_to_usermode_loop+0x1dd/0x1f0 [ 7676.292579] do_syscall_64+0x3d6/0x440 [ 7676.294130] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7676.300152] RIP: 0033:0x7f1a63983cd8 [ 7676.302800] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7676.318937] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7676.327365] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7676.332971] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7676.339329] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7676.344190] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7676.348291] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7688.058610] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 7688.061461] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 7688.064649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7688.064728] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7688.064736] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7688.064739] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7688.064752] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7688.064756] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7688.064823] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7688.064826] RDX: 0000000000000ced RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7688.064828] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7688.064831] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7688.064833] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7688.064837] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7688.064840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7688.064843] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7688.064851] Call Trace: [ 7688.064856] [ 7688.064861] ? show_regs.cold.9+0x22/0x2f [ 7688.064870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7688.064876] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7688.064880] ? watchdog+0x40/0x40 [ 7688.064884] ? hrtimer_interrupt+0x128/0x2c0 [ 7688.064888] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7688.065656] ? apic_timer_interrupt+0xf/0x20 [ 7688.065663] [ 7688.065666] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7688.065671] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7688.065676] do_raw_spin_lock+0x177/0x190 [ 7688.065682] _raw_spin_lock+0x21/0x30 [ 7688.065688] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7688.066300] obd_mod_free+0x46/0x140 [ptlrpc] [ 7688.071163] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7688.071241] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7688.071248] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7688.071251] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7688.071266] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7688.071270] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7688.071274] RAX: 0000000000005a06 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7688.071276] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7688.071278] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7688.071280] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7688.071283] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7688.071286] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7688.071289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7688.071292] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7688.071300] Call Trace: [ 7688.071305] [ 7688.071311] ? show_regs.cold.9+0x22/0x2f [ 7688.071327] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7688.071334] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7688.071339] ? watchdog+0x40/0x40 [ 7688.071483] ? hrtimer_interrupt+0x128/0x2c0 [ 7688.071491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7688.071498] ? apic_timer_interrupt+0xf/0x20 [ 7688.071503] [ 7688.071509] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7688.071516] do_raw_spin_lock+0x177/0x190 [ 7688.071520] _raw_spin_lock+0x21/0x30 [ 7688.071525] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7688.071888] ? do_raw_spin_unlock+0x75/0x190 [ 7688.071895] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7688.072286] ? do_wait_intr+0xf0/0xf0 [ 7688.072295] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7688.072687] mdc_close+0x53b/0xd70 [mdc] [ 7688.072854] lmv_close+0x18d/0x490 [lmv] [ 7688.072888] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7688.073033] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7688.073131] ll_md_close+0x274/0x800 [lustre] [ 7688.073230] ll_file_release+0x128/0xc50 [lustre] [ 7688.073323] __fput+0x10d/0x440 [ 7688.073333] ____fput+0x12/0x20 [ 7688.073338] task_work_run+0xc6/0x110 [ 7688.073363] exit_to_usermode_loop+0x1dd/0x1f0 [ 7688.073370] do_syscall_64+0x3d6/0x440 [ 7688.073374] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7688.073382] RIP: 0033:0x7f1a63983cd8 [ 7688.073389] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7688.073393] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7688.073398] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7688.073614] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7688.073616] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7688.073619] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7688.073626] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7688.704875] mdc_commit_open+0x91/0xb0 [mdc] [ 7688.717889] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7688.734283] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7688.740233] after_reply+0xe2c/0x1780 [ptlrpc] [ 7688.749991] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7688.754972] ? __next_timer_interrupt+0x160/0x160 [ 7688.756722] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7688.768387] ? do_wait_intr+0xf0/0xf0 [ 7688.769757] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7688.792411] kthread+0x1d1/0x200 [ 7688.793612] ? set_kthread_struct+0x70/0x70 [ 7688.804503] ret_from_fork+0x1f/0x30 [ 7700.055204] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 7700.059867] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7700.136766] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7700.150933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7700.158555] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7700.166296] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7700.189401] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7700.197021] RAX: 0000000000007920 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7700.205506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7700.214210] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7700.223000] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7700.232870] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7700.241567] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7700.249673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7700.253707] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7700.262899] Call Trace: [ 7700.266025] [ 7700.267729] ? show_regs.cold.9+0x22/0x2f [ 7700.271645] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7700.278524] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7700.281009] ? watchdog+0x40/0x40 [ 7700.287515] ? hrtimer_interrupt+0x128/0x2c0 [ 7700.290513] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7700.294370] ? apic_timer_interrupt+0xf/0x20 [ 7700.302023] [ 7700.308858] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7700.323533] do_raw_spin_lock+0x177/0x190 [ 7700.328942] _raw_spin_lock+0x21/0x30 [ 7700.336509] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7700.344435] ? do_raw_spin_unlock+0x75/0x190 [ 7700.354764] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7700.360374] ? do_wait_intr+0xf0/0xf0 [ 7700.361961] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7700.370272] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7700.375314] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7700.381311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7700.388984] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7700.395903] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7700.398615] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7700.402586] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7700.408941] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7700.414849] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7700.420504] lmv_intent_open+0x338/0x1150 [lmv] [ 7700.423317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7700.433027] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7700.438699] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7700.441053] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7700.449450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7700.457360] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7700.464356] ? do_raw_spin_unlock+0x75/0x190 [ 7700.468212] ? _raw_spin_unlock+0x12/0x30 [ 7700.471318] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7700.477224] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7700.481413] ll_atomic_open+0x36a/0xd70 [lustre] [ 7700.485305] lookup_open+0x356/0xb40 [ 7700.493883] do_last+0x313/0xfc0 [ 7700.495135] ? nd_jump_root+0xe5/0x160 [ 7700.500131] ? path_init+0x437/0x520 [ 7700.503428] path_openat+0xf7/0x500 [ 7700.507112] ? do_raw_spin_unlock+0x75/0x190 [ 7700.510791] do_filp_open+0x99/0x140 [ 7700.517806] ? getname_flags+0x6e/0x330 [ 7700.521900] ? __check_object_size+0xff/0x256 [ 7700.525654] ? do_raw_spin_unlock+0x75/0x190 [ 7700.529419] ? _raw_spin_unlock+0x12/0x30 [ 7700.533564] do_sys_openat2+0x2b4/0x410 [ 7700.536923] do_sys_open+0x73/0xa0 [ 7700.543814] __x64_sys_openat+0x24/0x30 [ 7700.546124] do_syscall_64+0xc1/0x440 [ 7700.549800] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7700.554145] RIP: 0033:0x7f1a63983332 [ 7700.559345] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7700.575825] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7700.585523] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7700.591537] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7700.597954] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7700.604465] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7700.613865] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7703.521235] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5680s! [ 7703.524911] Showing busy workqueues and worker pools: [ 7703.527947] workqueue events: flags=0x0 [ 7703.529766] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7703.533043] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7703.536839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7703.540008] pending: sec_gc_main [ptlrpc] [ 7703.542474] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7703.546218] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7703.551525] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7703.555277] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7703.558800] workqueue events_long: flags=0x0 [ 7703.560568] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7703.564048] pending: lu_objects_flush [obdclass] [ 7703.566944] workqueue events_unbound: flags=0x2 [ 7703.568972] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7703.571669] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7703.576935] workqueue events_power_efficient: flags=0x80 [ 7703.579919] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7703.583384] pending: neigh_periodic_work, do_cache_clean [ 7703.586280] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7703.590395] pending: neigh_periodic_work, check_lifetime [ 7703.593061] workqueue mm_percpu_wq: flags=0x8 [ 7703.595402] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7703.598613] pending: vmstat_update [ 7703.600527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7703.603786] pending: vmstat_update [ 7703.605627] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7703.608855] pending: vmstat_update [ 7703.610727] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7703.614086] pending: vmstat_update [ 7703.615951] workqueue writeback: flags=0x4a [ 7703.617956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7703.621011] pending: wb_workfn, wb_workfn [ 7703.623391] workqueue xprtiod: flags=0xa [ 7703.625377] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7703.628254] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7703.632193] workqueue ipv6_addrconf: flags=0x40008 [ 7703.634729] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7703.637861] pending: addrconf_verify_work [ 7703.640034] workqueue knbd0-recv: flags=0x1a [ 7703.642478] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7703.645703] in-flight: 647:recv_work [ 7703.647653] workqueue ptlrpc_pinger: flags=0x2 [ 7703.649786] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7703.652638] pending: ptlrpc_pinger_main [ptlrpc] [ 7703.655012] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7693s workers=2 idle: 652 [ 7704.057308] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7704.070713] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7704.134405] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7704.152075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7704.168820] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7704.177909] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7704.204655] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7704.210677] RAX: 0000000000006c87 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7704.224209] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7704.230725] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7704.237536] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7704.250795] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7704.259008] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7704.277597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7704.293657] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7704.317400] Call Trace: [ 7704.318484] [ 7704.321591] ? show_regs.cold.9+0x22/0x2f [ 7704.336618] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7704.348873] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7704.350446] ? watchdog+0x40/0x40 [ 7704.360739] ? hrtimer_interrupt+0x128/0x2c0 [ 7704.362144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7704.379523] ? apic_timer_interrupt+0xf/0x20 [ 7704.381022] [ 7704.381706] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7704.396320] do_raw_spin_lock+0x177/0x190 [ 7704.397741] _raw_spin_lock+0x21/0x30 [ 7704.409023] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7704.417562] ? do_raw_spin_unlock+0x75/0x190 [ 7704.420653] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7704.423958] ? do_wait_intr+0xf0/0xf0 [ 7704.426563] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7704.431323] mdc_close+0x53b/0xd70 [mdc] [ 7704.432885] lmv_close+0x18d/0x490 [lmv] [ 7704.437120] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7704.445488] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7704.447067] ll_md_close+0x274/0x800 [lustre] [ 7704.450940] ll_file_release+0x128/0xc50 [lustre] [ 7704.455628] __fput+0x10d/0x440 [ 7704.457147] ____fput+0x12/0x20 [ 7704.458998] task_work_run+0xc6/0x110 [ 7704.461666] exit_to_usermode_loop+0x1dd/0x1f0 [ 7704.464681] do_syscall_64+0x3d6/0x440 [ 7704.471471] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7704.473394] RIP: 0033:0x7f1a63983cd8 [ 7704.481305] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7704.513636] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7704.521626] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7704.525327] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7704.531636] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7704.538839] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7704.546174] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7716.056054] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 7716.058028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 7716.060646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [ 7716.064361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7716.064444] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7716.064452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7716.064456] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7716.064475] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7716.064479] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7716.064483] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7716.064486] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7716.064492] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7716.064494] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7716.064497] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7716.064500] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7716.064503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7716.064506] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7716.064513] Call Trace: [ 7716.064522] [ 7716.064529] ? show_regs.cold.9+0x22/0x2f [ 7716.064545] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7716.064552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7716.064556] ? watchdog+0x40/0x40 [ 7716.064561] ? hrtimer_interrupt+0x128/0x2c0 [ 7716.064565] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7716.064573] ? apic_timer_interrupt+0xf/0x20 [ 7716.064578] [ 7716.064586] ? native_safe_halt+0x2e/0x50 [ 7716.064591] kvm_wait+0xcd/0xe0 [ 7716.064600] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7716.064607] do_raw_spin_lock+0x177/0x190 [ 7716.064616] _raw_spin_lock+0x21/0x30 [ 7716.064622] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7716.064955] obd_mod_free+0x46/0x140 [ptlrpc] [ 7716.065246] mdc_commit_open+0x91/0xb0 [mdc] [ 7716.065341] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7716.080798] libcfs(O) [ 7716.102314] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7716.111558] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7716.111603] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7716.111613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7716.111615] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7716.111636] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7716.115766] after_reply+0xe2c/0x1780 [ptlrpc] [ 7716.120647] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 [ 7716.128013] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7716.132067] ORIG_RAX: ffffffffffffff13 [ 7716.134808] ? __next_timer_interrupt+0x160/0x160 [ 7716.134831] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7716.138701] RAX: 0000000000007d82 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7716.138711] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7716.138716] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7716.141459] ? do_wait_intr+0xf0/0xf0 [ 7716.145478] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7716.145487] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7716.145491] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7716.145494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7716.145497] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7716.145504] Call Trace: [ 7716.145511] [ 7716.145516] ? show_regs.cold.9+0x22/0x2f [ 7716.145527] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7716.145534] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7716.145538] ? watchdog+0x40/0x40 [ 7716.145542] ? hrtimer_interrupt+0x128/0x2c0 [ 7716.145546] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7716.145554] ? apic_timer_interrupt+0xf/0x20 [ 7716.145558] [ 7716.145561] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7716.145569] do_raw_spin_lock+0x177/0x190 [ 7716.145573] _raw_spin_lock+0x21/0x30 [ 7716.145577] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7716.145853] ? do_raw_spin_unlock+0x75/0x190 [ 7716.145858] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7716.146052] ? do_wait_intr+0xf0/0xf0 [ 7716.146060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7716.149876] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7716.152654] mdc_close+0x53b/0xd70 [mdc] [ 7716.152703] lmv_close+0x18d/0x490 [lmv] [ 7716.155363] kthread+0x1d1/0x200 [ 7716.157519] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7716.157646] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7716.157725] ll_md_close+0x274/0x800 [lustre] [ 7716.157806] ll_file_release+0x128/0xc50 [lustre] [ 7716.157887] __fput+0x10d/0x440 [ 7716.157896] ____fput+0x12/0x20 [ 7716.157900] task_work_run+0xc6/0x110 [ 7716.157908] exit_to_usermode_loop+0x1dd/0x1f0 [ 7716.157916] do_syscall_64+0x3d6/0x440 [ 7716.157921] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7716.157931] RIP: 0033:0x7f1a63983cd8 [ 7716.157939] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7716.157943] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7716.157949] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7716.157952] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7716.157954] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7716.157956] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7716.157958] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7716.320622] ? set_kthread_struct+0x70/0x70 [ 7716.322338] ret_from_fork+0x1f/0x30 [ 7728.055871] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 7728.069859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7728.167251] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7728.193399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7728.215316] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7728.227410] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7728.267167] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7728.288970] RAX: 000000000000740a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7728.298823] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7728.311396] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7728.325597] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7728.347109] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7728.362939] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7728.378499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7728.387743] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7728.396085] Call Trace: [ 7728.400811] [ 7728.404239] ? show_regs.cold.9+0x22/0x2f [ 7728.414721] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7728.430398] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7728.436182] ? watchdog+0x40/0x40 [ 7728.437284] ? hrtimer_interrupt+0x128/0x2c0 [ 7728.450160] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7728.460571] ? apic_timer_interrupt+0xf/0x20 [ 7728.464616] [ 7728.465800] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7728.478218] do_raw_spin_lock+0x177/0x190 [ 7728.486255] _raw_spin_lock+0x21/0x30 [ 7728.490354] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7728.495628] ? do_raw_spin_unlock+0x75/0x190 [ 7728.497699] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7728.501137] ? do_wait_intr+0xf0/0xf0 [ 7728.504077] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7728.514154] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7728.516298] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7728.522768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7728.525148] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7728.528699] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7728.531245] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7728.540623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7728.545805] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7728.550776] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7728.554440] lmv_intent_open+0x338/0x1150 [lmv] [ 7728.559679] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7728.566508] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7728.569682] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7728.571936] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7728.576362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7728.585387] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7728.587330] ? do_raw_spin_unlock+0x75/0x190 [ 7728.593342] ? _raw_spin_unlock+0x12/0x30 [ 7728.595110] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7728.597707] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7728.602156] ll_atomic_open+0x36a/0xd70 [lustre] [ 7728.604739] lookup_open+0x356/0xb40 [ 7728.613740] do_last+0x313/0xfc0 [ 7728.615472] ? nd_jump_root+0xe5/0x160 [ 7728.620507] ? path_init+0x437/0x520 [ 7728.625043] path_openat+0xf7/0x500 [ 7728.629289] ? do_raw_spin_unlock+0x75/0x190 [ 7728.632590] do_filp_open+0x99/0x140 [ 7728.636122] ? getname_flags+0x6e/0x330 [ 7728.640301] ? __check_object_size+0xff/0x256 [ 7728.644631] ? do_raw_spin_unlock+0x75/0x190 [ 7728.646718] ? _raw_spin_unlock+0x12/0x30 [ 7728.649126] do_sys_openat2+0x2b4/0x410 [ 7728.650973] do_sys_open+0x73/0xa0 [ 7728.655615] __x64_sys_openat+0x24/0x30 [ 7728.657570] do_syscall_64+0xc1/0x440 [ 7728.659490] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7728.662143] RIP: 0033:0x7f1a63983332 [ 7728.664029] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7728.680343] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7728.685819] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7728.690654] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7728.693631] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7728.697523] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7728.701296] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7732.057019] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 7732.060460] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7732.081423] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7732.086323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7732.090144] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7732.092276] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7732.102261] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7732.104783] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7732.106907] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7732.109096] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7732.111499] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7732.117801] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7732.120417] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7732.124600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7732.126595] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7732.129709] Call Trace: [ 7732.130595] [ 7732.131463] ? show_regs.cold.9+0x22/0x2f [ 7732.133090] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7732.135450] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7732.137211] ? watchdog+0x40/0x40 [ 7732.138452] ? hrtimer_interrupt+0x128/0x2c0 [ 7732.148310] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7732.151495] ? apic_timer_interrupt+0xf/0x20 [ 7732.153560] [ 7732.154327] ? native_safe_halt+0x2e/0x50 [ 7732.157987] kvm_wait+0xcd/0xe0 [ 7732.160134] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7732.163890] do_raw_spin_lock+0x177/0x190 [ 7732.168122] _raw_spin_lock+0x21/0x30 [ 7732.169807] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7732.173691] ? do_raw_spin_unlock+0x75/0x190 [ 7732.177636] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7732.181191] ? do_wait_intr+0xf0/0xf0 [ 7732.183547] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7732.186259] mdc_close+0x53b/0xd70 [mdc] [ 7732.188081] lmv_close+0x18d/0x490 [lmv] [ 7732.189496] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7732.192095] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7732.194325] ll_md_close+0x274/0x800 [lustre] [ 7732.196567] ll_file_release+0x128/0xc50 [lustre] [ 7732.198982] __fput+0x10d/0x440 [ 7732.200494] ____fput+0x12/0x20 [ 7732.201992] task_work_run+0xc6/0x110 [ 7732.203731] exit_to_usermode_loop+0x1dd/0x1f0 [ 7732.205701] do_syscall_64+0x3d6/0x440 [ 7732.207396] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7732.209860] RIP: 0033:0x7f1a63983cd8 [ 7732.211468] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7732.219992] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7732.223433] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7732.227642] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7732.230893] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7732.233865] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7732.239245] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7733.730198] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5710s! [ 7733.739825] Showing busy workqueues and worker pools: [ 7733.746132] workqueue events: flags=0x0 [ 7733.751369] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7733.755500] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7733.758606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7733.761289] pending: sec_gc_main [ptlrpc] [ 7733.763360] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7733.769344] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7733.775033] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7733.776855] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7733.787359] workqueue events_long: flags=0x0 [ 7733.789304] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7733.792485] pending: lu_objects_flush [obdclass] [ 7733.800348] workqueue events_unbound: flags=0x2 [ 7733.803195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7733.810354] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7733.815382] workqueue events_power_efficient: flags=0x80 [ 7733.818038] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7733.821193] pending: neigh_periodic_work, do_cache_clean [ 7733.825242] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7733.828390] pending: neigh_periodic_work, check_lifetime [ 7733.831579] workqueue mm_percpu_wq: flags=0x8 [ 7733.837705] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7733.846650] pending: vmstat_update [ 7733.850240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7733.853256] pending: vmstat_update [ 7733.854458] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7733.857248] pending: vmstat_update [ 7733.858440] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7733.861206] pending: vmstat_update [ 7733.862397] workqueue writeback: flags=0x4a [ 7733.864206] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7733.866442] pending: wb_workfn, wb_workfn [ 7733.867774] workqueue xprtiod: flags=0xa [ 7733.869061] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7733.871021] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7733.873425] workqueue ipv6_addrconf: flags=0x40008 [ 7733.875087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7733.877131] pending: addrconf_verify_work [ 7733.878413] workqueue knbd0-recv: flags=0x1a [ 7733.880103] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7733.882087] in-flight: 647:recv_work [ 7733.883310] workqueue ptlrpc_pinger: flags=0x2 [ 7733.885031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7733.887290] pending: ptlrpc_pinger_main [ptlrpc] [ 7733.896442] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7723s workers=2 idle: 652 [ 7744.058015] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 7744.060464] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 7744.060977] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7744.061126] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7744.061132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7744.061135] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7744.061159] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7744.061163] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7744.061168] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7744.061171] RDX: 0000000000003d8b RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7744.061173] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7744.061176] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7744.061179] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7744.061182] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7744.061185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7744.061187] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7744.061195] Call Trace: [ 7744.061211] [ 7744.061268] ? show_regs.cold.9+0x22/0x2f [ 7744.061279] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7744.061286] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7744.061289] ? watchdog+0x40/0x40 [ 7744.061294] ? hrtimer_interrupt+0x128/0x2c0 [ 7744.061297] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7744.061305] ? apic_timer_interrupt+0xf/0x20 [ 7744.061309] [ 7744.061312] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7744.061400] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7744.061406] do_raw_spin_lock+0x177/0x190 [ 7744.061411] _raw_spin_lock+0x21/0x30 [ 7744.061416] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7744.061680] obd_mod_free+0x46/0x140 [ptlrpc] [ 7744.061940] mdc_commit_open+0x91/0xb0 [mdc] [ 7744.061978] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7744.062163] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7744.062435] after_reply+0xe2c/0x1780 [ptlrpc] [ 7744.062628] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7744.062901] ? __next_timer_interrupt+0x160/0x160 [ 7744.062909] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7744.063107] ? do_wait_intr+0xf0/0xf0 [ 7744.063115] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7744.063375] kthread+0x1d1/0x200 [ 7744.063381] ? set_kthread_struct+0x70/0x70 [ 7744.063384] ret_from_fork+0x1f/0x30 [ 7744.240174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7744.274699] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7744.283760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7744.290063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7744.292659] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7744.311135] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7744.316135] RAX: 0000000000003b22 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7744.320064] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7744.324177] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7744.333025] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7744.336617] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7744.340626] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7744.344599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7744.352125] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7744.356652] Call Trace: [ 7744.358560] [ 7744.359496] ? show_regs.cold.9+0x22/0x2f [ 7744.362128] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7744.364493] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7744.366299] ? watchdog+0x40/0x40 [ 7744.368305] ? hrtimer_interrupt+0x128/0x2c0 [ 7744.370151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7744.371936] ? apic_timer_interrupt+0xf/0x20 [ 7744.373779] [ 7744.374560] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7744.376681] do_raw_spin_lock+0x177/0x190 [ 7744.378209] _raw_spin_lock+0x21/0x30 [ 7744.379532] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7744.382383] ? do_raw_spin_unlock+0x75/0x190 [ 7744.383994] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7744.385869] ? do_wait_intr+0xf0/0xf0 [ 7744.387032] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7744.389577] mdc_close+0x53b/0xd70 [mdc] [ 7744.390971] lmv_close+0x18d/0x490 [lmv] [ 7744.392267] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7744.394601] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7744.396146] ll_md_close+0x274/0x800 [lustre] [ 7744.397488] ll_file_release+0x128/0xc50 [lustre] [ 7744.399099] __fput+0x10d/0x440 [ 7744.400225] ____fput+0x12/0x20 [ 7744.401260] task_work_run+0xc6/0x110 [ 7744.403604] exit_to_usermode_loop+0x1dd/0x1f0 [ 7744.406270] do_syscall_64+0x3d6/0x440 [ 7744.408322] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7744.410721] RIP: 0033:0x7f1a63983cd8 [ 7744.416727] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7744.429385] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7744.433512] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7744.436773] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7744.441024] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7744.450258] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7744.455651] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7760.053064] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 7760.055924] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7760.057827] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 7760.073389] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7760.073404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7760.073408] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7760.073423] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7760.073427] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7760.073434] RAX: 0000000000006f85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7760.073437] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7760.073439] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7760.073441] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7760.073443] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7760.073446] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7760.073449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7760.073451] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7760.073460] Call Trace: [ 7760.073465] [ 7760.073471] ? show_regs.cold.9+0x22/0x2f [ 7760.073482] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7760.073488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7760.073491] ? watchdog+0x40/0x40 [ 7760.073496] ? hrtimer_interrupt+0x128/0x2c0 [ 7760.073500] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7760.073508] ? apic_timer_interrupt+0xf/0x20 [ 7760.073513] [ 7760.073515] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7760.073523] do_raw_spin_lock+0x177/0x190 [ 7760.073528] _raw_spin_lock+0x21/0x30 [ 7760.073533] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7760.073790] ? do_raw_spin_unlock+0x75/0x190 [ 7760.073794] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7760.073918] ? do_wait_intr+0xf0/0xf0 [ 7760.073924] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7760.074076] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7760.074263] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7760.074483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7760.074576] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7760.074607] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7760.074626] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7760.074699] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7760.074772] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7760.074924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7760.088390] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7760.088484] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7760.088496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7760.088499] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7760.088513] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7760.088517] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7760.088521] RAX: 0000000000007f85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7760.088523] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7760.088525] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7760.088527] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7760.088529] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7760.088532] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7760.088535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7760.088538] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7760.088546] Call Trace: [ 7760.088555] [ 7760.088561] ? show_regs.cold.9+0x22/0x2f [ 7760.088569] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7760.088581] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7760.088584] ? watchdog+0x40/0x40 [ 7760.088589] ? hrtimer_interrupt+0x128/0x2c0 [ 7760.088592] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7760.088601] ? apic_timer_interrupt+0xf/0x20 [ 7760.088606] [ 7760.088609] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7760.088615] do_raw_spin_lock+0x177/0x190 [ 7760.088620] _raw_spin_lock+0x21/0x30 [ 7760.088625] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7760.088982] ? do_raw_spin_unlock+0x75/0x190 [ 7760.088988] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7760.089259] ? do_wait_intr+0xf0/0xf0 [ 7760.089266] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7760.089462] mdc_close+0x53b/0xd70 [mdc] [ 7760.089505] lmv_close+0x18d/0x490 [lmv] [ 7760.089536] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7760.089736] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7760.089873] ll_md_close+0x274/0x800 [lustre] [ 7760.090367] ll_file_release+0x128/0xc50 [lustre] [ 7760.090553] __fput+0x10d/0x440 [ 7760.090562] ____fput+0x12/0x20 [ 7760.090566] task_work_run+0xc6/0x110 [ 7760.090574] exit_to_usermode_loop+0x1dd/0x1f0 [ 7760.090582] do_syscall_64+0x3d6/0x440 [ 7760.090587] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7760.090595] RIP: 0033:0x7f1a63983cd8 [ 7760.090603] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7760.090607] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7760.090612] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7760.090614] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7760.090616] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7760.090618] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7760.090620] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7760.458636] lmv_intent_open+0x338/0x1150 [lmv] [ 7760.466419] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7760.468218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7760.471885] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7760.475974] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7760.481039] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7760.484997] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7760.487524] ? do_raw_spin_unlock+0x75/0x190 [ 7760.490030] ? _raw_spin_unlock+0x12/0x30 [ 7760.492918] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7760.504162] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7760.506408] ll_atomic_open+0x36a/0xd70 [lustre] [ 7760.510248] lookup_open+0x356/0xb40 [ 7760.513892] do_last+0x313/0xfc0 [ 7760.515090] ? nd_jump_root+0xe5/0x160 [ 7760.518348] ? path_init+0x437/0x520 [ 7760.521830] path_openat+0xf7/0x500 [ 7760.524080] ? do_raw_spin_unlock+0x75/0x190 [ 7760.529420] do_filp_open+0x99/0x140 [ 7760.530356] ? getname_flags+0x6e/0x330 [ 7760.533079] ? __check_object_size+0xff/0x256 [ 7760.536403] ? do_raw_spin_unlock+0x75/0x190 [ 7760.539104] ? _raw_spin_unlock+0x12/0x30 [ 7760.540900] do_sys_openat2+0x2b4/0x410 [ 7760.544631] do_sys_open+0x73/0xa0 [ 7760.546129] __x64_sys_openat+0x24/0x30 [ 7760.549028] do_syscall_64+0xc1/0x440 [ 7760.552308] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7760.555300] RIP: 0033:0x7f1a63983332 [ 7760.556721] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7760.565398] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7760.567552] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7760.578207] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7760.583740] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7760.586258] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7760.593587] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7763.936094] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5740s! [ 7763.940259] Showing busy workqueues and worker pools: [ 7763.945262] workqueue events: flags=0x0 [ 7763.950094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7763.956162] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7763.961518] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7763.969339] pending: sec_gc_main [ptlrpc] [ 7763.975379] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7763.978327] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7763.987069] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7763.989972] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7763.993674] workqueue events_long: flags=0x0 [ 7763.997330] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7764.001039] pending: lu_objects_flush [obdclass] [ 7764.005368] workqueue events_unbound: flags=0x2 [ 7764.007515] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7764.024861] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7764.037433] workqueue events_power_efficient: flags=0x80 [ 7764.042282] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7764.052969] pending: neigh_periodic_work, do_cache_clean [ 7764.055144] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7764.064477] pending: neigh_periodic_work, check_lifetime [ 7764.070517] workqueue mm_percpu_wq: flags=0x8 [ 7764.072434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7764.076177] pending: vmstat_update [ 7764.080562] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7764.083434] pending: vmstat_update [ 7764.086944] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7764.090981] pending: vmstat_update [ 7764.092649] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7764.099025] pending: vmstat_update [ 7764.101030] workqueue writeback: flags=0x4a [ 7764.103484] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7764.106462] pending: wb_workfn, wb_workfn [ 7764.109322] workqueue xprtiod: flags=0xa [ 7764.111336] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7764.114126] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7764.120728] workqueue ipv6_addrconf: flags=0x40008 [ 7764.122360] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7764.130252] pending: addrconf_verify_work [ 7764.134131] workqueue knbd0-recv: flags=0x1a [ 7764.138421] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7764.141746] in-flight: 647:recv_work [ 7764.144195] workqueue ptlrpc_pinger: flags=0x2 [ 7764.146041] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7764.148918] pending: ptlrpc_pinger_main [ptlrpc] [ 7764.153021] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7754s workers=2 idle: 652 [ 7772.056020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 7772.058330] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7772.060342] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 7772.087198] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7772.087215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7772.087218] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7772.087231] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7772.087235] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7772.087241] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7772.087243] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7772.087246] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7772.087248] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7772.087250] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7772.087257] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7772.087260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7772.087263] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7772.087270] Call Trace: [ 7772.087276] [ 7772.087281] ? show_regs.cold.9+0x22/0x2f [ 7772.087290] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7772.087297] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7772.087306] ? watchdog+0x40/0x40 [ 7772.087310] ? hrtimer_interrupt+0x128/0x2c0 [ 7772.087314] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7772.087324] ? apic_timer_interrupt+0xf/0x20 [ 7772.087329] [ 7772.087334] ? native_safe_halt+0x2e/0x50 [ 7772.087339] kvm_wait+0xcd/0xe0 [ 7772.087346] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7772.087355] do_raw_spin_lock+0x177/0x190 [ 7772.087360] _raw_spin_lock+0x21/0x30 [ 7772.087364] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7772.093322] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7772.093400] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7772.093408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7772.093411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7772.093433] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7772.093437] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7772.093441] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7772.093444] RDX: 00000000000010d3 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7772.093446] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7772.093448] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7772.093450] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7772.093453] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7772.093456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7772.093458] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7772.093465] Call Trace: [ 7772.093470] [ 7772.093476] ? show_regs.cold.9+0x22/0x2f [ 7772.093486] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7772.093492] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7772.093496] ? watchdog+0x40/0x40 [ 7772.093499] ? hrtimer_interrupt+0x128/0x2c0 [ 7772.093502] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7772.093511] ? apic_timer_interrupt+0xf/0x20 [ 7772.093515] [ 7772.093517] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7772.093521] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7772.093525] do_raw_spin_lock+0x177/0x190 [ 7772.093529] _raw_spin_lock+0x21/0x30 [ 7772.093551] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7772.102157] ? do_raw_spin_unlock+0x75/0x190 [ 7772.102180] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7772.102464] ? do_wait_intr+0xf0/0xf0 [ 7772.102473] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7772.102943] mdc_close+0x53b/0xd70 [mdc] [ 7772.102987] lmv_close+0x18d/0x490 [lmv] [ 7772.103077] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7772.103192] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7772.103617] ll_md_close+0x274/0x800 [lustre] [ 7772.110647] obd_mod_free+0x46/0x140 [ptlrpc] [ 7772.116362] ll_file_release+0x128/0xc50 [lustre] [ 7772.116482] __fput+0x10d/0x440 [ 7772.116493] ____fput+0x12/0x20 [ 7772.116497] task_work_run+0xc6/0x110 [ 7772.116506] exit_to_usermode_loop+0x1dd/0x1f0 [ 7772.116513] do_syscall_64+0x3d6/0x440 [ 7772.116517] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7772.116526] RIP: 0033:0x7f1a63983cd8 [ 7772.116533] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7772.116547] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7772.116553] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7772.116555] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7772.116557] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7772.116559] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7772.116561] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7772.347928] mdc_commit_open+0x91/0xb0 [mdc] [ 7772.350288] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7772.355068] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7772.359980] after_reply+0xe2c/0x1780 [ptlrpc] [ 7772.362674] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7772.365256] ? __next_timer_interrupt+0x160/0x160 [ 7772.367546] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7772.369980] ? do_wait_intr+0xf0/0xf0 [ 7772.372099] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7772.374979] kthread+0x1d1/0x200 [ 7772.377279] ? set_kthread_struct+0x70/0x70 [ 7772.379590] ret_from_fork+0x1f/0x30 [ 7788.057162] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 7788.062689] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7788.092832] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7788.099468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7788.105715] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7788.109501] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7788.121441] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7788.125143] RAX: 0000000000007720 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7788.129339] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7788.133317] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7788.137365] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7788.141882] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7788.145841] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7788.153657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7788.160475] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7788.164781] Call Trace: [ 7788.165949] [ 7788.166669] ? show_regs.cold.9+0x22/0x2f [ 7788.170052] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7788.173270] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7788.176039] ? watchdog+0x40/0x40 [ 7788.177677] ? hrtimer_interrupt+0x128/0x2c0 [ 7788.180366] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7788.182668] ? apic_timer_interrupt+0xf/0x20 [ 7788.184787] [ 7788.185777] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7788.187883] do_raw_spin_lock+0x177/0x190 [ 7788.189604] _raw_spin_lock+0x21/0x30 [ 7788.191361] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7788.194049] ? do_raw_spin_unlock+0x75/0x190 [ 7788.196136] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7788.198125] ? do_wait_intr+0xf0/0xf0 [ 7788.199744] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7788.202653] mdc_close+0x53b/0xd70 [mdc] [ 7788.204024] lmv_close+0x18d/0x490 [lmv] [ 7788.206033] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7788.211293] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7788.213600] ll_md_close+0x274/0x800 [lustre] [ 7788.215254] ll_file_release+0x128/0xc50 [lustre] [ 7788.217008] __fput+0x10d/0x440 [ 7788.218148] ____fput+0x12/0x20 [ 7788.219532] task_work_run+0xc6/0x110 [ 7788.221047] exit_to_usermode_loop+0x1dd/0x1f0 [ 7788.223535] do_syscall_64+0x3d6/0x440 [ 7788.225089] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7788.227561] RIP: 0033:0x7f1a63983cd8 [ 7788.228776] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7788.234847] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7788.238115] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7788.240727] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7788.243447] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7788.246605] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7788.252946] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7792.053266] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7792.062878] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7792.086492] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7792.091927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7792.096187] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7792.099481] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7792.107291] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7792.111001] RAX: 000000000000622a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7792.114502] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7792.117131] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7792.120067] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7792.123243] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7792.126305] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7792.130304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7792.133750] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7792.137137] Call Trace: [ 7792.138995] [ 7792.139940] ? show_regs.cold.9+0x22/0x2f [ 7792.141747] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7792.143720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7792.145649] ? watchdog+0x40/0x40 [ 7792.147002] ? hrtimer_interrupt+0x128/0x2c0 [ 7792.149045] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7792.151674] ? apic_timer_interrupt+0xf/0x20 [ 7792.153708] [ 7792.154772] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7792.157422] do_raw_spin_lock+0x177/0x190 [ 7792.159135] _raw_spin_lock+0x21/0x30 [ 7792.160983] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7792.164104] ? do_raw_spin_unlock+0x75/0x190 [ 7792.165724] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7792.168308] ? do_wait_intr+0xf0/0xf0 [ 7792.170036] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7792.172697] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7792.174819] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7792.176989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7792.179697] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7792.182632] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7792.184820] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7792.186907] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7792.189992] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7792.192994] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7792.195998] lmv_intent_open+0x338/0x1150 [lmv] [ 7792.198155] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7792.200867] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7792.203528] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7792.205768] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7792.208179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7792.210780] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7792.212756] ? do_raw_spin_unlock+0x75/0x190 [ 7792.215059] ? _raw_spin_unlock+0x12/0x30 [ 7792.216550] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7792.218866] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7792.221088] ll_atomic_open+0x36a/0xd70 [lustre] [ 7792.222799] lookup_open+0x356/0xb40 [ 7792.224288] do_last+0x313/0xfc0 [ 7792.225817] ? nd_jump_root+0xe5/0x160 [ 7792.227723] ? path_init+0x437/0x520 [ 7792.229875] path_openat+0xf7/0x500 [ 7792.231725] ? do_raw_spin_unlock+0x75/0x190 [ 7792.233883] do_filp_open+0x99/0x140 [ 7792.235718] ? getname_flags+0x6e/0x330 [ 7792.237818] ? __check_object_size+0xff/0x256 [ 7792.240152] ? do_raw_spin_unlock+0x75/0x190 [ 7792.242135] ? _raw_spin_unlock+0x12/0x30 [ 7792.244050] do_sys_openat2+0x2b4/0x410 [ 7792.245997] do_sys_open+0x73/0xa0 [ 7792.248016] __x64_sys_openat+0x24/0x30 [ 7792.249921] do_syscall_64+0xc1/0x440 [ 7792.251674] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7792.253909] RIP: 0033:0x7f1a63983332 [ 7792.255725] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7792.263284] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7792.265755] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7792.268026] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7792.270276] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7792.272717] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7792.274821] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7794.656652] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5771s! [ 7794.659020] Showing busy workqueues and worker pools: [ 7794.660950] workqueue events: flags=0x0 [ 7794.663849] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7794.672028] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7794.678435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7794.686007] pending: sec_gc_main [ptlrpc] [ 7794.688047] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7794.692224] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7794.696911] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7794.701781] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7794.708112] workqueue events_long: flags=0x0 [ 7794.713607] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7794.716142] pending: lu_objects_flush [obdclass] [ 7794.718107] workqueue events_unbound: flags=0x2 [ 7794.720751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7794.723560] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7794.730047] workqueue events_power_efficient: flags=0x80 [ 7794.732108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7794.734943] pending: neigh_periodic_work, do_cache_clean [ 7794.737257] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7794.741656] pending: neigh_periodic_work, check_lifetime [ 7794.743552] workqueue mm_percpu_wq: flags=0x8 [ 7794.745365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7794.747041] pending: vmstat_update [ 7794.747978] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7794.753250] pending: vmstat_update [ 7794.754874] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7794.757011] pending: vmstat_update [ 7794.758124] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7794.760106] pending: vmstat_update [ 7794.761169] workqueue writeback: flags=0x4a [ 7794.762723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7794.766620] pending: wb_workfn, wb_workfn [ 7794.768761] workqueue xprtiod: flags=0xa [ 7794.771082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7794.774600] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7794.779131] workqueue ipv6_addrconf: flags=0x40008 [ 7794.781885] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7794.785160] pending: addrconf_verify_work [ 7794.787518] workqueue knbd0-recv: flags=0x1a [ 7794.793847] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7794.802214] in-flight: 647:recv_work [ 7794.806899] workqueue ptlrpc_pinger: flags=0x2 [ 7794.811089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7794.816957] pending: ptlrpc_pinger_main [ptlrpc] [ 7794.824540] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7784s workers=2 idle: 652 [ 7800.056048] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7800.058021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7800.061660] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7800.063160] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7800.063169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7800.063172] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7800.063185] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7800.063189] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7800.063194] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7800.063196] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7800.063198] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7800.063200] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7800.063202] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7800.063205] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7800.063208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7800.063210] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7800.063223] Call Trace: [ 7800.063230] [ 7800.063236] ? show_regs.cold.9+0x22/0x2f [ 7800.063246] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7800.063253] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7800.063258] ? watchdog+0x40/0x40 [ 7800.063265] ? hrtimer_interrupt+0x128/0x2c0 [ 7800.063282] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7800.063288] ? apic_timer_interrupt+0xf/0x20 [ 7800.063292] [ 7800.063295] ? native_safe_halt+0x2e/0x50 [ 7800.063298] kvm_wait+0xcd/0xe0 [ 7800.063305] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7800.063313] do_raw_spin_lock+0x177/0x190 [ 7800.063317] _raw_spin_lock+0x21/0x30 [ 7800.063321] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7800.063705] ? do_raw_spin_unlock+0x75/0x190 [ 7800.063713] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7800.063943] ? do_wait_intr+0xf0/0xf0 [ 7800.078278] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7800.078378] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7800.078390] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7800.078393] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7800.078415] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7800.078420] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7800.078428] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7800.078431] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7800.078433] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7800.078435] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7800.078437] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7800.078440] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7800.078446] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7800.078448] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7800.078457] Call Trace: [ 7800.078463] [ 7800.078468] ? show_regs.cold.9+0x22/0x2f [ 7800.078479] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7800.078485] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7800.078489] ? watchdog+0x40/0x40 [ 7800.078494] ? hrtimer_interrupt+0x128/0x2c0 [ 7800.078498] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7800.078505] ? apic_timer_interrupt+0xf/0x20 [ 7800.078516] [ 7800.078519] ? native_safe_halt+0x2e/0x50 [ 7800.078523] kvm_wait+0xcd/0xe0 [ 7800.078530] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7800.078538] do_raw_spin_lock+0x177/0x190 [ 7800.078543] _raw_spin_lock+0x21/0x30 [ 7800.078548] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7800.113079] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7800.123743] obd_mod_free+0x46/0x140 [ptlrpc] [ 7800.124024] mdc_commit_open+0x91/0xb0 [mdc] [ 7800.124061] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7800.124252] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7800.124453] after_reply+0xe2c/0x1780 [ptlrpc] [ 7800.124645] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7800.124838] ? __next_timer_interrupt+0x160/0x160 [ 7800.124847] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7800.125046] ? do_wait_intr+0xf0/0xf0 [ 7800.125056] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7800.125248] kthread+0x1d1/0x200 [ 7800.125254] ? set_kthread_struct+0x70/0x70 [ 7800.125257] ret_from_fork+0x1f/0x30 [ 7800.435531] mdc_close+0x53b/0xd70 [mdc] [ 7800.437191] lmv_close+0x18d/0x490 [lmv] [ 7800.438651] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7800.440363] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7800.441768] ll_md_close+0x274/0x800 [lustre] [ 7800.443203] ll_file_release+0x128/0xc50 [lustre] [ 7800.444650] __fput+0x10d/0x440 [ 7800.445549] ____fput+0x12/0x20 [ 7800.446569] task_work_run+0xc6/0x110 [ 7800.447744] exit_to_usermode_loop+0x1dd/0x1f0 [ 7800.449031] do_syscall_64+0x3d6/0x440 [ 7800.449973] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7800.451227] RIP: 0033:0x7f1a63983cd8 [ 7800.452346] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7800.457792] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7800.459934] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7800.461848] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7800.463769] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7800.465883] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7800.467776] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7816.057052] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 7816.059840] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7816.083256] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7816.087734] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7816.092046] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7816.094762] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7816.102695] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7816.105836] RAX: 0000000000005810 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7816.108819] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7816.111683] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7816.114032] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7816.116932] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7816.119591] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7816.122499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7816.124471] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7816.127040] Call Trace: [ 7816.128030] [ 7816.128771] ? show_regs.cold.9+0x22/0x2f [ 7816.130219] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7816.132742] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7816.134542] ? watchdog+0x40/0x40 [ 7816.135731] ? hrtimer_interrupt+0x128/0x2c0 [ 7816.137425] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7816.139305] ? apic_timer_interrupt+0xf/0x20 [ 7816.140882] [ 7816.142402] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7816.144523] do_raw_spin_lock+0x177/0x190 [ 7816.146208] _raw_spin_lock+0x21/0x30 [ 7816.147545] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7816.149748] ? do_raw_spin_unlock+0x75/0x190 [ 7816.151434] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7816.153501] ? do_wait_intr+0xf0/0xf0 [ 7816.155050] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7816.157203] mdc_close+0x53b/0xd70 [mdc] [ 7816.158699] lmv_close+0x18d/0x490 [lmv] [ 7816.160424] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7816.162687] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7816.164958] ll_md_close+0x274/0x800 [lustre] [ 7816.166770] ll_file_release+0x128/0xc50 [lustre] [ 7816.168668] __fput+0x10d/0x440 [ 7816.169782] ____fput+0x12/0x20 [ 7816.170928] task_work_run+0xc6/0x110 [ 7816.172232] exit_to_usermode_loop+0x1dd/0x1f0 [ 7816.173751] do_syscall_64+0x3d6/0x440 [ 7816.174973] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7816.176751] RIP: 0033:0x7f1a63983cd8 [ 7816.178089] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7816.184501] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7816.187077] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7816.189655] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7816.192091] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7816.194714] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7816.197331] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7820.057046] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7820.059915] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7820.082404] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7820.086884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7820.089875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7820.092905] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7820.101386] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7820.104624] RAX: 0000000000000406 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7820.109306] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7820.116905] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7820.119655] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7820.122055] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7820.123954] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7820.128322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7820.130785] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7820.133939] Call Trace: [ 7820.135193] [ 7820.135942] ? show_regs.cold.9+0x22/0x2f [ 7820.137601] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7820.139153] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7820.140822] ? watchdog+0x40/0x40 [ 7820.142292] ? hrtimer_interrupt+0x128/0x2c0 [ 7820.143713] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7820.145168] ? apic_timer_interrupt+0xf/0x20 [ 7820.146852] [ 7820.147651] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7820.149367] do_raw_spin_lock+0x177/0x190 [ 7820.150715] _raw_spin_lock+0x21/0x30 [ 7820.151813] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7820.153815] ? do_raw_spin_unlock+0x75/0x190 [ 7820.155557] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7820.157538] ? do_wait_intr+0xf0/0xf0 [ 7820.158704] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7820.160860] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7820.162660] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7820.164784] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7820.166662] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7820.168317] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7820.169741] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7820.171133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7820.172835] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7820.174833] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7820.176946] lmv_intent_open+0x338/0x1150 [lmv] [ 7820.178596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7820.180670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7820.182663] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7820.184255] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7820.185880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7820.187671] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7820.189214] ? do_raw_spin_unlock+0x75/0x190 [ 7820.190431] ? _raw_spin_unlock+0x12/0x30 [ 7820.191713] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7820.193936] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7820.195798] ll_atomic_open+0x36a/0xd70 [lustre] [ 7820.197748] lookup_open+0x356/0xb40 [ 7820.199018] do_last+0x313/0xfc0 [ 7820.204875] ? nd_jump_root+0xe5/0x160 [ 7820.208594] ? path_init+0x437/0x520 [ 7820.211927] path_openat+0xf7/0x500 [ 7820.216350] ? do_raw_spin_unlock+0x75/0x190 [ 7820.218908] do_filp_open+0x99/0x140 [ 7820.221927] ? getname_flags+0x6e/0x330 [ 7820.225328] ? __check_object_size+0xff/0x256 [ 7820.229899] ? do_raw_spin_unlock+0x75/0x190 [ 7820.232174] ? _raw_spin_unlock+0x12/0x30 [ 7820.236347] do_sys_openat2+0x2b4/0x410 [ 7820.238438] do_sys_open+0x73/0xa0 [ 7820.239625] __x64_sys_openat+0x24/0x30 [ 7820.241195] do_syscall_64+0xc1/0x440 [ 7820.244304] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7820.251761] RIP: 0033:0x7f1a63983332 [ 7820.254925] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7820.270386] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7820.280670] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7820.289466] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7820.293259] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7820.297203] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7820.300746] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7824.864141] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5801s! [ 7824.866466] Showing busy workqueues and worker pools: [ 7824.868380] workqueue events: flags=0x0 [ 7824.870638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7824.876202] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7824.883866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7824.895928] pending: sec_gc_main [ptlrpc] [ 7824.902429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7824.911583] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7824.919105] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7824.929614] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7824.937557] workqueue events_long: flags=0x0 [ 7824.941198] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7824.947012] pending: lu_objects_flush [obdclass] [ 7824.952420] workqueue events_unbound: flags=0x2 [ 7824.960093] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7824.971809] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7824.983848] workqueue events_power_efficient: flags=0x80 [ 7824.986879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7824.995198] pending: neigh_periodic_work, do_cache_clean [ 7825.013328] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7825.015086] pending: neigh_periodic_work, check_lifetime [ 7825.020492] workqueue mm_percpu_wq: flags=0x8 [ 7825.026487] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7825.029817] pending: vmstat_update [ 7825.032288] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7825.035500] pending: vmstat_update [ 7825.037185] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7825.040967] pending: vmstat_update [ 7825.042855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7825.047933] pending: vmstat_update [ 7825.050771] workqueue writeback: flags=0x4a [ 7825.055790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7825.061441] pending: wb_workfn, wb_workfn [ 7825.065521] workqueue xprtiod: flags=0xa [ 7825.067325] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7825.070839] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7825.073719] workqueue ipv6_addrconf: flags=0x40008 [ 7825.076333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7825.079103] pending: addrconf_verify_work [ 7825.080833] workqueue knbd0-recv: flags=0x1a [ 7825.083794] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7825.086352] in-flight: 647:recv_work [ 7825.087516] workqueue ptlrpc_pinger: flags=0x2 [ 7825.092715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7825.096457] pending: ptlrpc_pinger_main [ptlrpc] [ 7825.101563] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7815s workers=2 idle: 652 [ 7828.057861] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7828.058069] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7828.071220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7828.071365] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7828.071372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7828.071374] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7828.071537] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7828.071542] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7828.071547] RAX: 0000000000007e16 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7828.071550] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7828.071552] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7828.071554] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7828.071556] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7828.071559] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7828.071561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7828.071564] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7828.071572] Call Trace: [ 7828.071577] [ 7828.071582] ? show_regs.cold.9+0x22/0x2f [ 7828.071592] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7828.071599] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7828.071603] ? watchdog+0x40/0x40 [ 7828.071608] ? hrtimer_interrupt+0x128/0x2c0 [ 7828.071612] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7828.071621] ? apic_timer_interrupt+0xf/0x20 [ 7828.071625] [ 7828.071628] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7828.071634] do_raw_spin_lock+0x177/0x190 [ 7828.071638] _raw_spin_lock+0x21/0x30 [ 7828.071643] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7828.072243] ? do_raw_spin_unlock+0x75/0x190 [ 7828.072249] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7828.073189] ? do_wait_intr+0xf0/0xf0 [ 7828.073200] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7828.073701] mdc_close+0x53b/0xd70 [mdc] [ 7828.073738] lmv_close+0x18d/0x490 [lmv] [ 7828.073772] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7828.074193] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7828.074286] ll_md_close+0x274/0x800 [lustre] [ 7828.074369] ll_file_release+0x128/0xc50 [lustre] [ 7828.074462] __fput+0x10d/0x440 [ 7828.074472] ____fput+0x12/0x20 [ 7828.074476] task_work_run+0xc6/0x110 [ 7828.074483] exit_to_usermode_loop+0x1dd/0x1f0 [ 7828.074492] do_syscall_64+0x3d6/0x440 [ 7828.074496] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7828.074505] RIP: 0033:0x7f1a63983cd8 [ 7828.074513] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7828.074517] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7828.074521] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7828.074523] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7828.074525] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7828.074527] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7828.074529] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7828.308818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7828.370714] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7828.380349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7828.388516] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7828.391461] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7828.401283] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7828.411516] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7828.416676] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7828.422470] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7828.425588] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7828.428462] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7828.431213] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7828.434226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7828.436409] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7828.439230] Call Trace: [ 7828.440287] [ 7828.441204] ? show_regs.cold.9+0x22/0x2f [ 7828.442768] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7828.444771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7828.446623] ? watchdog+0x40/0x40 [ 7828.447836] ? hrtimer_interrupt+0x128/0x2c0 [ 7828.449593] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7828.451581] ? apic_timer_interrupt+0xf/0x20 [ 7828.453396] [ 7828.454299] ? native_safe_halt+0x2e/0x50 [ 7828.456072] kvm_wait+0xcd/0xe0 [ 7828.457321] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7828.459353] do_raw_spin_lock+0x177/0x190 [ 7828.460830] _raw_spin_lock+0x21/0x30 [ 7828.462497] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7828.464851] obd_mod_free+0x46/0x140 [ptlrpc] [ 7828.467050] mdc_commit_open+0x91/0xb0 [mdc] [ 7828.468729] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7828.471310] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7828.473807] after_reply+0xe2c/0x1780 [ptlrpc] [ 7828.475806] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7828.478233] ? __next_timer_interrupt+0x160/0x160 [ 7828.480078] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7828.481965] ? do_wait_intr+0xf0/0xf0 [ 7828.483508] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7828.486672] kthread+0x1d1/0x200 [ 7828.489333] ? set_kthread_struct+0x70/0x70 [ 7828.491265] ret_from_fork+0x1f/0x30 [ 7842.555015] rcu: INFO: rcu_sched self-detected stall on CPU [ 7842.556820] rcu: 3-...!: (3655777 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 7842.561960] (t=5820155 jiffies g=269633 q=2164) [ 7842.564189] rcu: rcu_sched kthread starved for 5819801 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 7842.569411] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7842.573731] rcu: RCU grace-period kthread stack dump: [ 7842.576569] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7842.583031] Call Trace: [ 7842.584544] __schedule+0x351/0xcb0 [ 7842.590339] schedule+0xc0/0x180 [ 7842.594214] schedule_timeout+0xb4/0x190 [ 7842.597712] ? do_raw_spin_unlock+0x75/0x190 [ 7842.603576] ? __next_timer_interrupt+0x160/0x160 [ 7842.610835] ? prepare_to_swait_event+0xf3/0x140 [ 7842.612565] rcu_gp_kthread+0x8dd/0x15b0 [ 7842.614818] ? force_qs_rnp+0x2c0/0x2c0 [ 7842.617514] kthread+0x1d1/0x200 [ 7842.621445] ? set_kthread_struct+0x70/0x70 [ 7842.624069] ret_from_fork+0x1f/0x30 [ 7842.626699] rcu: Stack dump where RCU GP kthread last ran: [ 7842.630656] NMI backtrace for cpu 3 [ 7842.635407] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7842.653415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7842.658869] Call Trace: [ 7842.660389] [ 7842.661827] ? dump_stack+0xbb/0x10e [ 7842.663949] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7842.674225] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7842.678488] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7842.683546] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7842.685478] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7842.690706] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 7842.695461] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 7842.705616] ? update_process_times+0x6d/0xc0 [ 7842.709588] ? tick_sched_handle.isra.18+0x27/0x90 [ 7842.713214] ? tick_sched_timer+0x53/0xb0 [ 7842.716839] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7842.723684] ? tick_sched_handle.isra.18+0x90/0x90 [ 7842.730010] ? hrtimer_interrupt+0x128/0x2c0 [ 7842.733958] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7842.737896] ? apic_timer_interrupt+0xf/0x20 [ 7842.740501] [ 7842.743690] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7842.748001] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7842.754363] ? do_raw_spin_lock+0x177/0x190 [ 7842.760229] ? _raw_spin_lock+0x21/0x30 [ 7842.769158] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7842.774224] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7842.780719] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7842.784795] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7842.791835] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7842.793592] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7842.796245] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7842.799149] ? __next_timer_interrupt+0x160/0x160 [ 7842.801256] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7842.806639] ? do_wait_intr+0xf0/0xf0 [ 7842.808909] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7842.811943] ? kthread+0x1d1/0x200 [ 7842.818189] ? set_kthread_struct+0x70/0x70 [ 7842.819726] ? ret_from_fork+0x1f/0x30 [ 7842.825748] NMI backtrace for cpu 3 [ 7842.827059] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7842.833401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7842.842433] Call Trace: [ 7842.846470] [ 7842.848383] ? dump_stack+0xbb/0x10e [ 7842.851677] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7842.854607] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7842.858822] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7842.866102] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7842.870505] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7842.874783] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7842.878017] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7842.879830] ? update_process_times+0x6d/0xc0 [ 7842.884014] ? tick_sched_handle.isra.18+0x27/0x90 [ 7842.887306] ? tick_sched_timer+0x53/0xb0 [ 7842.892672] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7842.896726] ? tick_sched_handle.isra.18+0x90/0x90 [ 7842.902874] ? hrtimer_interrupt+0x128/0x2c0 [ 7842.904529] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7842.912633] ? apic_timer_interrupt+0xf/0x20 [ 7842.914538] [ 7842.915796] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7842.920644] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7842.924203] ? do_raw_spin_lock+0x177/0x190 [ 7842.926912] ? _raw_spin_lock+0x21/0x30 [ 7842.929109] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7842.935393] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7842.939794] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7842.944208] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7842.947497] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7842.953708] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7842.960665] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7842.966653] ? __next_timer_interrupt+0x160/0x160 [ 7842.974002] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7842.977064] ? do_wait_intr+0xf0/0xf0 [ 7842.978264] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7842.980838] ? kthread+0x1d1/0x200 [ 7842.982245] ? set_kthread_struct+0x70/0x70 [ 7842.985191] ? ret_from_fork+0x1f/0x30 [ 7844.057371] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 7844.078518] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7844.118073] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7844.135556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7844.145996] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7844.157560] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7844.171895] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7844.176828] RAX: 0000000000005b5e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7844.184570] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7844.190614] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7844.211051] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7844.214085] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7844.217549] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7844.223499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7844.229529] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7844.238162] Call Trace: [ 7844.239699] [ 7844.240727] ? show_regs.cold.9+0x22/0x2f [ 7844.246463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7844.251293] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7844.260398] ? watchdog+0x40/0x40 [ 7844.264101] ? hrtimer_interrupt+0x128/0x2c0 [ 7844.273405] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7844.285539] ? apic_timer_interrupt+0xf/0x20 [ 7844.287204] [ 7844.287885] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7844.303559] do_raw_spin_lock+0x177/0x190 [ 7844.305036] _raw_spin_lock+0x21/0x30 [ 7844.322183] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7844.325581] ? do_raw_spin_unlock+0x75/0x190 [ 7844.327317] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7844.329482] ? do_wait_intr+0xf0/0xf0 [ 7844.331136] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7844.333962] mdc_close+0x53b/0xd70 [mdc] [ 7844.339248] lmv_close+0x18d/0x490 [lmv] [ 7844.341480] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7844.343316] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7844.347715] ll_md_close+0x274/0x800 [lustre] [ 7844.351027] ll_file_release+0x128/0xc50 [lustre] [ 7844.357077] __fput+0x10d/0x440 [ 7844.359963] ____fput+0x12/0x20 [ 7844.370396] task_work_run+0xc6/0x110 [ 7844.377351] exit_to_usermode_loop+0x1dd/0x1f0 [ 7844.381834] do_syscall_64+0x3d6/0x440 [ 7844.387527] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7844.391123] RIP: 0033:0x7f1a63983cd8 [ 7844.393266] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7844.402165] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7844.414034] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7844.424116] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7844.429241] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7844.435019] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7844.439004] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7852.053787] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7852.064388] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7852.113414] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7852.118158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7852.124847] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7852.126284] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7852.144745] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7852.155056] RAX: 0000000000006011 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7852.157642] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7852.165745] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7852.170159] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7852.172948] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7852.176506] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7852.180818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7852.184243] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7852.187241] Call Trace: [ 7852.188253] [ 7852.189187] ? show_regs.cold.9+0x22/0x2f [ 7852.191342] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7852.193185] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7852.194676] ? watchdog+0x40/0x40 [ 7852.195933] ? hrtimer_interrupt+0x128/0x2c0 [ 7852.197434] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7852.199164] ? apic_timer_interrupt+0xf/0x20 [ 7852.200522] [ 7852.201276] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7852.204513] do_raw_spin_lock+0x177/0x190 [ 7852.206671] _raw_spin_lock+0x21/0x30 [ 7852.208065] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7852.212123] ? do_raw_spin_unlock+0x75/0x190 [ 7852.213464] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7852.215604] ? do_wait_intr+0xf0/0xf0 [ 7852.217435] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7852.219865] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7852.221196] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7852.224673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7852.227721] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7852.229705] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7852.230857] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7852.231901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7852.238662] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7852.240291] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7852.241691] lmv_intent_open+0x338/0x1150 [lmv] [ 7852.251217] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7852.256041] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7852.258075] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7852.260149] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7852.264465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7852.269988] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7852.272405] ? do_raw_spin_unlock+0x75/0x190 [ 7852.274409] ? _raw_spin_unlock+0x12/0x30 [ 7852.277356] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7852.283968] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7852.285737] ll_atomic_open+0x36a/0xd70 [lustre] [ 7852.289362] lookup_open+0x356/0xb40 [ 7852.291162] do_last+0x313/0xfc0 [ 7852.294587] ? nd_jump_root+0xe5/0x160 [ 7852.301200] ? path_init+0x437/0x520 [ 7852.304344] path_openat+0xf7/0x500 [ 7852.307580] ? do_raw_spin_unlock+0x75/0x190 [ 7852.309700] do_filp_open+0x99/0x140 [ 7852.315158] ? getname_flags+0x6e/0x330 [ 7852.316538] ? __check_object_size+0xff/0x256 [ 7852.321986] ? do_raw_spin_unlock+0x75/0x190 [ 7852.325357] ? _raw_spin_unlock+0x12/0x30 [ 7852.329024] do_sys_openat2+0x2b4/0x410 [ 7852.332069] do_sys_open+0x73/0xa0 [ 7852.333829] __x64_sys_openat+0x24/0x30 [ 7852.335697] do_syscall_64+0xc1/0x440 [ 7852.337208] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7852.341885] RIP: 0033:0x7f1a63983332 [ 7852.344991] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7852.359798] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7852.362051] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7852.365370] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7852.368711] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7852.372046] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7852.375357] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7855.584085] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5832s! [ 7855.589059] Showing busy workqueues and worker pools: [ 7855.597241] workqueue events: flags=0x0 [ 7855.607662] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7855.610077] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7855.622456] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7855.625546] pending: sec_gc_main [ptlrpc] [ 7855.628474] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7855.630855] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7855.649195] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7855.656250] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7855.662227] workqueue events_long: flags=0x0 [ 7855.664095] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7855.668086] pending: lu_objects_flush [obdclass] [ 7855.671118] workqueue events_unbound: flags=0x2 [ 7855.673355] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7855.683946] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7855.696220] workqueue events_power_efficient: flags=0x80 [ 7855.700198] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7855.706321] pending: neigh_periodic_work, do_cache_clean [ 7855.708419] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7855.719364] pending: neigh_periodic_work, check_lifetime [ 7855.722496] workqueue mm_percpu_wq: flags=0x8 [ 7855.725125] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7855.732946] pending: vmstat_update [ 7855.735009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7855.738545] pending: vmstat_update [ 7855.744107] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7855.754507] pending: vmstat_update [ 7855.758593] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7855.763348] pending: vmstat_update [ 7855.768574] workqueue writeback: flags=0x4a [ 7855.771057] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7855.780616] pending: wb_workfn, wb_workfn [ 7855.782363] workqueue xprtiod: flags=0xa [ 7855.793104] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7855.804987] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7855.815351] workqueue ipv6_addrconf: flags=0x40008 [ 7855.818389] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7855.823138] pending: addrconf_verify_work [ 7855.825657] workqueue knbd0-recv: flags=0x1a [ 7855.827071] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7855.832556] in-flight: 647:recv_work [ 7855.839074] workqueue ptlrpc_pinger: flags=0x2 [ 7855.843248] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7855.853045] pending: ptlrpc_pinger_main [ptlrpc] [ 7855.861953] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7845s workers=2 idle: 652 [ 7856.056237] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7856.059604] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7856.081168] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7856.086634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7856.090975] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7856.092982] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7856.102354] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7856.105935] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7856.109409] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7856.113045] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7856.116487] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7856.120035] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7856.123893] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7856.127506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7856.129958] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7856.132940] Call Trace: [ 7856.133927] [ 7856.134605] ? show_regs.cold.9+0x22/0x2f [ 7856.136503] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7856.138474] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7856.141399] ? watchdog+0x40/0x40 [ 7856.142969] ? hrtimer_interrupt+0x128/0x2c0 [ 7856.144349] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7856.146414] ? apic_timer_interrupt+0xf/0x20 [ 7856.148640] [ 7856.149485] ? native_safe_halt+0x2e/0x50 [ 7856.151121] kvm_wait+0xcd/0xe0 [ 7856.152286] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7856.154579] do_raw_spin_lock+0x177/0x190 [ 7856.156515] _raw_spin_lock+0x21/0x30 [ 7856.158094] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7856.160960] ? do_raw_spin_unlock+0x75/0x190 [ 7856.162877] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7856.165185] ? do_wait_intr+0xf0/0xf0 [ 7856.166606] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7856.169148] mdc_close+0x53b/0xd70 [mdc] [ 7856.170627] lmv_close+0x18d/0x490 [lmv] [ 7856.172218] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7856.174563] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7856.176562] ll_md_close+0x274/0x800 [lustre] [ 7856.178524] ll_file_release+0x128/0xc50 [lustre] [ 7856.180374] __fput+0x10d/0x440 [ 7856.181594] ____fput+0x12/0x20 [ 7856.182967] task_work_run+0xc6/0x110 [ 7856.184258] exit_to_usermode_loop+0x1dd/0x1f0 [ 7856.185930] do_syscall_64+0x3d6/0x440 [ 7856.187370] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7856.189477] RIP: 0033:0x7f1a63983cd8 [ 7856.190958] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7856.197353] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7856.200063] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7856.202610] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7856.205107] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7856.207446] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7856.210088] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7868.058616] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7868.067806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7868.120517] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7868.129030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7868.135947] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7868.137080] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7868.146862] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7868.148633] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7868.151227] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7868.154362] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7868.156937] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7868.160580] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7868.163853] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7868.167542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7868.169698] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7868.173689] Call Trace: [ 7868.174999] [ 7868.176252] ? show_regs.cold.9+0x22/0x2f [ 7868.178287] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7868.180690] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7868.182829] ? watchdog+0x40/0x40 [ 7868.184491] ? hrtimer_interrupt+0x128/0x2c0 [ 7868.186551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7868.188964] ? apic_timer_interrupt+0xf/0x20 [ 7868.191044] [ 7868.192178] ? native_safe_halt+0x2e/0x50 [ 7868.193827] kvm_wait+0xcd/0xe0 [ 7868.195450] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7868.201589] do_raw_spin_lock+0x177/0x190 [ 7868.203665] _raw_spin_lock+0x21/0x30 [ 7868.205452] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7868.208073] obd_mod_free+0x46/0x140 [ptlrpc] [ 7868.210519] mdc_commit_open+0x91/0xb0 [mdc] [ 7868.212218] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7868.214055] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7868.216903] after_reply+0xe2c/0x1780 [ptlrpc] [ 7868.219212] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7868.220593] ? __next_timer_interrupt+0x160/0x160 [ 7868.225257] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7868.228783] ? do_wait_intr+0xf0/0xf0 [ 7868.230773] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7868.233815] kthread+0x1d1/0x200 [ 7868.235053] ? set_kthread_struct+0x70/0x70 [ 7868.236785] ret_from_fork+0x1f/0x30 [ 7872.057021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7872.060030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7872.085640] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7872.090014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7872.096438] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7872.098449] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7872.109922] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7872.115524] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7872.121432] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7872.126101] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7872.128500] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7872.133074] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7872.135664] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7872.139718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7872.141754] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7872.147326] Call Trace: [ 7872.149985] [ 7872.151054] ? show_regs.cold.9+0x22/0x2f [ 7872.153128] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7872.157447] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7872.159073] ? watchdog+0x40/0x40 [ 7872.160179] ? hrtimer_interrupt+0x128/0x2c0 [ 7872.163958] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7872.166606] ? apic_timer_interrupt+0xf/0x20 [ 7872.168958] [ 7872.170080] ? native_safe_halt+0x2e/0x50 [ 7872.175550] kvm_wait+0xcd/0xe0 [ 7872.177123] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7872.181110] do_raw_spin_lock+0x177/0x190 [ 7872.182942] _raw_spin_lock+0x21/0x30 [ 7872.184495] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7872.189326] ? do_raw_spin_unlock+0x75/0x190 [ 7872.191127] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7872.195437] ? do_wait_intr+0xf0/0xf0 [ 7872.197677] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7872.200486] mdc_close+0x53b/0xd70 [mdc] [ 7872.202684] lmv_close+0x18d/0x490 [lmv] [ 7872.207132] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7872.209190] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7872.211248] ll_md_close+0x274/0x800 [lustre] [ 7872.213659] ll_file_release+0x128/0xc50 [lustre] [ 7872.215682] __fput+0x10d/0x440 [ 7872.217989] ____fput+0x12/0x20 [ 7872.219432] task_work_run+0xc6/0x110 [ 7872.223469] exit_to_usermode_loop+0x1dd/0x1f0 [ 7872.225938] do_syscall_64+0x3d6/0x440 [ 7872.228725] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7872.235005] RIP: 0033:0x7f1a63983cd8 [ 7872.237385] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7872.246844] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7872.253268] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7872.255709] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7872.257778] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7872.263439] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7872.267528] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7880.053435] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7880.060931] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7880.110985] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7880.124815] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7880.141970] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7880.150335] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7880.174148] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7880.181872] RAX: 000000000000750b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7880.190784] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7880.197852] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7880.200479] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7880.206978] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7880.213044] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7880.218513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7880.222517] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7880.227178] Call Trace: [ 7880.228606] [ 7880.230705] ? show_regs.cold.9+0x22/0x2f [ 7880.241448] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7880.251173] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7880.256167] ? watchdog+0x40/0x40 [ 7880.261603] ? hrtimer_interrupt+0x128/0x2c0 [ 7880.268594] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7880.272166] ? apic_timer_interrupt+0xf/0x20 [ 7880.276886] [ 7880.278823] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7880.282614] do_raw_spin_lock+0x177/0x190 [ 7880.285217] _raw_spin_lock+0x21/0x30 [ 7880.287662] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7880.290223] ? do_raw_spin_unlock+0x75/0x190 [ 7880.292641] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7880.295231] ? do_wait_intr+0xf0/0xf0 [ 7880.296923] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7880.299463] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7880.302462] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7880.305336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7880.307905] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7880.310426] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7880.312934] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7880.315266] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7880.317862] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7880.321009] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7880.326381] lmv_intent_open+0x338/0x1150 [lmv] [ 7880.329290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7880.332821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7880.335973] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7880.338406] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7880.340724] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7880.343759] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7880.346403] ? do_raw_spin_unlock+0x75/0x190 [ 7880.349730] ? _raw_spin_unlock+0x12/0x30 [ 7880.352840] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7880.355810] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7880.358679] ll_atomic_open+0x36a/0xd70 [lustre] [ 7880.362306] lookup_open+0x356/0xb40 [ 7880.364988] do_last+0x313/0xfc0 [ 7880.367394] ? nd_jump_root+0xe5/0x160 [ 7880.369685] ? path_init+0x437/0x520 [ 7880.371598] path_openat+0xf7/0x500 [ 7880.373443] ? do_raw_spin_unlock+0x75/0x190 [ 7880.375905] do_filp_open+0x99/0x140 [ 7880.377962] ? getname_flags+0x6e/0x330 [ 7880.380634] ? __check_object_size+0xff/0x256 [ 7880.383138] ? do_raw_spin_unlock+0x75/0x190 [ 7880.385840] ? _raw_spin_unlock+0x12/0x30 [ 7880.387813] do_sys_openat2+0x2b4/0x410 [ 7880.389938] do_sys_open+0x73/0xa0 [ 7880.391901] __x64_sys_openat+0x24/0x30 [ 7880.394279] do_syscall_64+0xc1/0x440 [ 7880.396343] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7880.398751] RIP: 0033:0x7f1a63983332 [ 7880.400687] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7880.411001] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7880.415733] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7880.420837] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7880.424471] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7880.428623] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7880.433417] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7884.057664] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7884.062689] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7884.095187] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7884.103721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7884.110919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7884.119100] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7884.128809] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7884.134225] RAX: 0000000000000d0a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7884.141825] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7884.146433] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7884.156082] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7884.166006] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7884.172790] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7884.181087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7884.185800] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7884.191951] Call Trace: [ 7884.192923] [ 7884.193646] ? show_regs.cold.9+0x22/0x2f [ 7884.196739] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7884.200689] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7884.203666] ? watchdog+0x40/0x40 [ 7884.206930] ? hrtimer_interrupt+0x128/0x2c0 [ 7884.213921] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7884.225211] ? apic_timer_interrupt+0xf/0x20 [ 7884.226476] [ 7884.227146] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7884.232013] do_raw_spin_lock+0x177/0x190 [ 7884.234202] _raw_spin_lock+0x21/0x30 [ 7884.237186] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7884.244175] ? do_raw_spin_unlock+0x75/0x190 [ 7884.246345] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7884.250068] ? do_wait_intr+0xf0/0xf0 [ 7884.253362] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7884.258161] mdc_close+0x53b/0xd70 [mdc] [ 7884.260141] lmv_close+0x18d/0x490 [lmv] [ 7884.261567] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7884.265560] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7884.269340] ll_md_close+0x274/0x800 [lustre] [ 7884.277750] ll_file_release+0x128/0xc50 [lustre] [ 7884.279341] __fput+0x10d/0x440 [ 7884.280317] ____fput+0x12/0x20 [ 7884.284808] task_work_run+0xc6/0x110 [ 7884.288900] exit_to_usermode_loop+0x1dd/0x1f0 [ 7884.291083] do_syscall_64+0x3d6/0x440 [ 7884.293759] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7884.300752] RIP: 0033:0x7f1a63983cd8 [ 7884.303542] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7884.320890] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7884.327546] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7884.335843] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7884.342705] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7884.347915] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7884.351999] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7886.306251] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5863s! [ 7886.312135] Showing busy workqueues and worker pools: [ 7886.316847] workqueue events: flags=0x0 [ 7886.321335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7886.326022] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7886.333878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7886.342072] pending: sec_gc_main [ptlrpc] [ 7886.350551] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7886.357899] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7886.372015] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7886.374403] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7886.382367] workqueue events_long: flags=0x0 [ 7886.385772] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7886.390400] pending: lu_objects_flush [obdclass] [ 7886.394919] workqueue events_unbound: flags=0x2 [ 7886.397817] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7886.401381] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7886.407808] workqueue events_power_efficient: flags=0x80 [ 7886.410090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7886.414003] pending: neigh_periodic_work, do_cache_clean [ 7886.420532] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7886.424557] pending: neigh_periodic_work, check_lifetime [ 7886.428750] workqueue mm_percpu_wq: flags=0x8 [ 7886.431661] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7886.436269] pending: vmstat_update [ 7886.438560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7886.442947] pending: vmstat_update [ 7886.445247] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7886.452721] pending: vmstat_update [ 7886.455994] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7886.471625] pending: vmstat_update [ 7886.478610] workqueue writeback: flags=0x4a [ 7886.487100] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7886.491630] pending: wb_workfn, wb_workfn [ 7886.494700] workqueue xprtiod: flags=0xa [ 7886.497283] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7886.501444] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7886.507050] workqueue ipv6_addrconf: flags=0x40008 [ 7886.512070] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7886.518600] pending: addrconf_verify_work [ 7886.521903] workqueue knbd0-recv: flags=0x1a [ 7886.525524] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7886.529650] in-flight: 647:recv_work [ 7886.532939] workqueue ptlrpc_pinger: flags=0x2 [ 7886.535740] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7886.539527] pending: ptlrpc_pinger_main [ptlrpc] [ 7886.542736] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7876s workers=2 idle: 652 [ 7896.058011] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7896.065095] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7896.094312] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7896.102500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7896.108120] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7896.111126] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7896.120316] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7896.123646] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7896.126010] RDX: 0000000000003812 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7896.129881] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7896.133718] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7896.137663] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7896.141030] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7896.145288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7896.147977] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7896.151807] Call Trace: [ 7896.153117] [ 7896.154133] ? show_regs.cold.9+0x22/0x2f [ 7896.156185] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7896.158318] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7896.160496] ? watchdog+0x40/0x40 [ 7896.162127] ? hrtimer_interrupt+0x128/0x2c0 [ 7896.164183] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7896.166731] ? apic_timer_interrupt+0xf/0x20 [ 7896.169120] [ 7896.170063] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7896.174740] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7896.177302] do_raw_spin_lock+0x177/0x190 [ 7896.181133] _raw_spin_lock+0x21/0x30 [ 7896.183944] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7896.187586] obd_mod_free+0x46/0x140 [ptlrpc] [ 7896.191082] mdc_commit_open+0x91/0xb0 [mdc] [ 7896.194463] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7896.201508] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7896.204357] after_reply+0xe2c/0x1780 [ptlrpc] [ 7896.207906] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7896.210624] ? __next_timer_interrupt+0x160/0x160 [ 7896.212725] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7896.214937] ? do_wait_intr+0xf0/0xf0 [ 7896.216534] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7896.219044] kthread+0x1d1/0x200 [ 7896.220533] ? set_kthread_struct+0x70/0x70 [ 7896.222726] ret_from_fork+0x1f/0x30 [ 7900.057029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7900.061344] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7900.082979] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7900.087476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7900.090540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7900.093093] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7900.099246] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7900.101276] RAX: 0000000000003b9c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7900.103773] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7900.106037] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7900.107922] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7900.110094] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7900.112544] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7900.115870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7900.117878] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7900.120587] Call Trace: [ 7900.121479] [ 7900.122086] ? show_regs.cold.9+0x22/0x2f [ 7900.123840] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7900.125784] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7900.127484] ? watchdog+0x40/0x40 [ 7900.128576] ? hrtimer_interrupt+0x128/0x2c0 [ 7900.129977] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7900.131783] ? apic_timer_interrupt+0xf/0x20 [ 7900.133448] [ 7900.134083] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7900.136477] do_raw_spin_lock+0x177/0x190 [ 7900.137820] _raw_spin_lock+0x21/0x30 [ 7900.139146] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7900.142582] ? do_raw_spin_unlock+0x75/0x190 [ 7900.144148] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7900.146626] ? do_wait_intr+0xf0/0xf0 [ 7900.147966] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7900.150097] mdc_close+0x53b/0xd70 [mdc] [ 7900.151666] lmv_close+0x18d/0x490 [lmv] [ 7900.153103] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7900.155791] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7900.157777] ll_md_close+0x274/0x800 [lustre] [ 7900.159812] ll_file_release+0x128/0xc50 [lustre] [ 7900.161612] __fput+0x10d/0x440 [ 7900.162868] ____fput+0x12/0x20 [ 7900.164102] task_work_run+0xc6/0x110 [ 7900.165804] exit_to_usermode_loop+0x1dd/0x1f0 [ 7900.167942] do_syscall_64+0x3d6/0x440 [ 7900.169502] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7900.171876] RIP: 0033:0x7f1a63983cd8 [ 7900.173274] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7900.180085] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7900.182946] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7900.185614] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7900.188025] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7900.190411] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7900.192645] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7912.053197] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7912.056306] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7912.059537] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7912.059690] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7912.059698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7912.059702] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7912.059717] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7912.059722] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7912.059726] RAX: 0000000000007f98 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7912.059728] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7912.059731] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7912.059733] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7912.059735] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7912.059738] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7912.059741] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7912.059742] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7912.059751] Call Trace: [ 7912.059756] [ 7912.059762] ? show_regs.cold.9+0x22/0x2f [ 7912.059771] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7912.059777] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7912.059781] ? watchdog+0x40/0x40 [ 7912.059784] ? hrtimer_interrupt+0x128/0x2c0 [ 7912.059788] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7912.059797] ? apic_timer_interrupt+0xf/0x20 [ 7912.059801] [ 7912.059804] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7912.059810] do_raw_spin_lock+0x177/0x190 [ 7912.059815] _raw_spin_lock+0x21/0x30 [ 7912.059820] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7912.066443] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7912.066875] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7912.066882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7912.066884] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7912.066899] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7912.066902] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7912.066907] RAX: 000000000000591d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7912.066909] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7912.066911] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7912.066913] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7912.066916] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7912.066919] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7912.066922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7912.066924] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7912.066931] Call Trace: [ 7912.066936] [ 7912.066941] ? show_regs.cold.9+0x22/0x2f [ 7912.066950] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7912.066955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7912.066959] ? watchdog+0x40/0x40 [ 7912.066963] ? hrtimer_interrupt+0x128/0x2c0 [ 7912.066966] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7912.066974] ? apic_timer_interrupt+0xf/0x20 [ 7912.066978] [ 7912.066980] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7912.066986] do_raw_spin_lock+0x177/0x190 [ 7912.066990] _raw_spin_lock+0x21/0x30 [ 7912.066994] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7912.067708] ? do_raw_spin_unlock+0x75/0x190 [ 7912.067714] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7912.114591] ? do_raw_spin_unlock+0x75/0x190 [ 7912.114622] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7912.114898] ? do_wait_intr+0xf0/0xf0 [ 7912.114906] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7912.121122] ? do_wait_intr+0xf0/0xf0 [ 7912.121141] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7912.122251] mdc_close+0x53b/0xd70 [mdc] [ 7912.122288] lmv_close+0x18d/0x490 [lmv] [ 7912.122312] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7912.122416] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7912.123309] ll_md_close+0x274/0x800 [lustre] [ 7912.123379] ll_file_release+0x128/0xc50 [lustre] [ 7912.123587] __fput+0x10d/0x440 [ 7912.123597] ____fput+0x12/0x20 [ 7912.123601] task_work_run+0xc6/0x110 [ 7912.123612] exit_to_usermode_loop+0x1dd/0x1f0 [ 7912.123618] do_syscall_64+0x3d6/0x440 [ 7912.123622] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7912.123630] RIP: 0033:0x7f1a63983cd8 [ 7912.123638] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7912.123641] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7912.123646] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7912.123648] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7912.123650] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7912.123652] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7912.123655] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7912.700370] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7912.705590] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7912.711788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7912.714889] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7912.719015] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7912.721813] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7912.727744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7912.732085] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7912.739955] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7912.743137] lmv_intent_open+0x338/0x1150 [lmv] [ 7912.745584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7912.747842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7912.756545] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7912.761526] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7912.763697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7912.769963] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7912.772034] ? do_raw_spin_unlock+0x75/0x190 [ 7912.775308] ? _raw_spin_unlock+0x12/0x30 [ 7912.776547] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7912.777915] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7912.780992] ll_atomic_open+0x36a/0xd70 [lustre] [ 7912.784896] lookup_open+0x356/0xb40 [ 7912.790218] do_last+0x313/0xfc0 [ 7912.791034] ? nd_jump_root+0xe5/0x160 [ 7912.791986] ? path_init+0x437/0x520 [ 7912.794882] path_openat+0xf7/0x500 [ 7912.798085] ? do_raw_spin_unlock+0x75/0x190 [ 7912.804457] do_filp_open+0x99/0x140 [ 7912.805777] ? getname_flags+0x6e/0x330 [ 7912.807857] ? __check_object_size+0xff/0x256 [ 7912.809428] ? do_raw_spin_unlock+0x75/0x190 [ 7912.811027] ? _raw_spin_unlock+0x12/0x30 [ 7912.817002] do_sys_openat2+0x2b4/0x410 [ 7912.818445] do_sys_open+0x73/0xa0 [ 7912.822141] __x64_sys_openat+0x24/0x30 [ 7912.825017] do_syscall_64+0xc1/0x440 [ 7912.837301] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7912.839216] RIP: 0033:0x7f1a63983332 [ 7912.843285] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7912.860588] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7912.865809] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7912.868139] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7912.874707] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7912.882860] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7912.896295] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7917.025062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5894s! [ 7917.028244] Showing busy workqueues and worker pools: [ 7917.033750] workqueue events: flags=0x0 [ 7917.036920] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7917.049524] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7917.061035] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7917.065426] pending: sec_gc_main [ptlrpc] [ 7917.068954] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7917.077683] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7917.089217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7917.094929] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7917.101906] workqueue events_long: flags=0x0 [ 7917.108469] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7917.118290] pending: lu_objects_flush [obdclass] [ 7917.129962] workqueue events_unbound: flags=0x2 [ 7917.135543] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7917.139904] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7917.150496] workqueue events_power_efficient: flags=0x80 [ 7917.153470] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7917.159038] pending: neigh_periodic_work, do_cache_clean [ 7917.164508] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7917.170682] pending: neigh_periodic_work, check_lifetime [ 7917.174364] workqueue mm_percpu_wq: flags=0x8 [ 7917.176434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7917.182596] pending: vmstat_update [ 7917.185587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7917.190902] pending: vmstat_update [ 7917.195057] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7917.197987] pending: vmstat_update [ 7917.204652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7917.209351] pending: vmstat_update [ 7917.214528] workqueue writeback: flags=0x4a [ 7917.219624] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7917.228010] pending: wb_workfn, wb_workfn [ 7917.230093] workqueue xprtiod: flags=0xa [ 7917.235341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7917.239619] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7917.243591] workqueue ipv6_addrconf: flags=0x40008 [ 7917.246402] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7917.249045] pending: addrconf_verify_work [ 7917.250913] workqueue knbd0-recv: flags=0x1a [ 7917.254046] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7917.256918] in-flight: 647:recv_work [ 7917.259041] workqueue ptlrpc_pinger: flags=0x2 [ 7917.261496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7917.265307] pending: ptlrpc_pinger_main [ptlrpc] [ 7917.269223] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7907s workers=2 idle: 652 [ 7924.059264] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7924.063452] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7924.084782] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7924.088522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7924.091576] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7924.093751] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7924.099171] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7924.101454] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7924.103381] RDX: 00000000000040eb RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7924.105181] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7924.115603] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7924.118429] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7924.121490] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7924.125475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7924.129059] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7924.131804] Call Trace: [ 7924.133296] [ 7924.134551] ? show_regs.cold.9+0x22/0x2f [ 7924.136162] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7924.139199] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7924.141222] ? watchdog+0x40/0x40 [ 7924.144051] ? hrtimer_interrupt+0x128/0x2c0 [ 7924.146170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7924.148700] ? apic_timer_interrupt+0xf/0x20 [ 7924.150895] [ 7924.151732] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7924.158553] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7924.162790] do_raw_spin_lock+0x177/0x190 [ 7924.165733] _raw_spin_lock+0x21/0x30 [ 7924.168513] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7924.175733] obd_mod_free+0x46/0x140 [ptlrpc] [ 7924.182742] mdc_commit_open+0x91/0xb0 [mdc] [ 7924.186198] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7924.193552] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7924.200647] after_reply+0xe2c/0x1780 [ptlrpc] [ 7924.202683] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7924.208561] ? __next_timer_interrupt+0x160/0x160 [ 7924.210925] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7924.213608] ? do_wait_intr+0xf0/0xf0 [ 7924.215720] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7924.222235] kthread+0x1d1/0x200 [ 7924.223646] ? set_kthread_struct+0x70/0x70 [ 7924.225567] ret_from_fork+0x1f/0x30 [ 7928.059694] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7928.066458] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7928.091441] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7928.096120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7928.100305] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7928.102420] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7928.108216] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7928.111176] RAX: 0000000000004bef RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7928.114432] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7928.122785] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7928.128640] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7928.131891] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7928.134863] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7928.138476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7928.140816] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7928.143607] Call Trace: [ 7928.147518] [ 7928.148481] ? show_regs.cold.9+0x22/0x2f [ 7928.149997] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7928.153546] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7928.155120] ? watchdog+0x40/0x40 [ 7928.156388] ? hrtimer_interrupt+0x128/0x2c0 [ 7928.158131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7928.159842] ? apic_timer_interrupt+0xf/0x20 [ 7928.161146] [ 7928.161774] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7928.163716] do_raw_spin_lock+0x177/0x190 [ 7928.166405] _raw_spin_lock+0x21/0x30 [ 7928.167498] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7928.172546] ? do_raw_spin_unlock+0x75/0x190 [ 7928.175733] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7928.179906] ? do_wait_intr+0xf0/0xf0 [ 7928.183168] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7928.189425] mdc_close+0x53b/0xd70 [mdc] [ 7928.195466] lmv_close+0x18d/0x490 [lmv] [ 7928.197407] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7928.199946] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7928.202221] ll_md_close+0x274/0x800 [lustre] [ 7928.203706] ll_file_release+0x128/0xc50 [lustre] [ 7928.205306] __fput+0x10d/0x440 [ 7928.206509] ____fput+0x12/0x20 [ 7928.211435] task_work_run+0xc6/0x110 [ 7928.213593] exit_to_usermode_loop+0x1dd/0x1f0 [ 7928.220607] do_syscall_64+0x3d6/0x440 [ 7928.224847] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7928.227698] RIP: 0033:0x7f1a63983cd8 [ 7928.229384] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7928.240743] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7928.248742] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7928.253088] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7928.256923] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7928.260032] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7928.263019] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7940.056021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7940.060264] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7940.108637] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7940.116572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7940.134922] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7940.144661] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7940.161856] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7940.168754] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7940.173139] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7940.176922] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7940.182878] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7940.188683] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7940.194835] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7940.201878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7940.211395] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7940.217239] Call Trace: [ 7940.218800] [ 7940.220556] ? show_regs.cold.9+0x22/0x2f [ 7940.223703] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7940.225478] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7940.233607] ? watchdog+0x40/0x40 [ 7940.237940] ? hrtimer_interrupt+0x128/0x2c0 [ 7940.243861] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7940.248661] ? apic_timer_interrupt+0xf/0x20 [ 7940.253776] [ 7940.256794] ? native_safe_halt+0x2e/0x50 [ 7940.257986] kvm_wait+0xcd/0xe0 [ 7940.258995] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7940.264313] do_raw_spin_lock+0x177/0x190 [ 7940.267755] _raw_spin_lock+0x21/0x30 [ 7940.269757] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7940.272638] ? do_raw_spin_unlock+0x75/0x190 [ 7940.277585] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7940.281778] ? do_wait_intr+0xf0/0xf0 [ 7940.284555] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7940.287736] mdc_close+0x53b/0xd70 [mdc] [ 7940.291402] lmv_close+0x18d/0x490 [lmv] [ 7940.294383] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7940.297516] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7940.304334] ll_md_close+0x274/0x800 [lustre] [ 7940.307354] ll_file_release+0x128/0xc50 [lustre] [ 7940.309634] __fput+0x10d/0x440 [ 7940.314107] ____fput+0x12/0x20 [ 7940.316684] task_work_run+0xc6/0x110 [ 7940.318571] exit_to_usermode_loop+0x1dd/0x1f0 [ 7940.322442] do_syscall_64+0x3d6/0x440 [ 7940.326405] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7940.332256] RIP: 0033:0x7f1a63983cd8 [ 7940.334747] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7940.347774] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7940.351598] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7940.355253] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7940.358556] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7940.361868] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7940.365611] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7944.053027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7944.062905] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7944.096303] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7944.111380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7944.118673] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7944.123491] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7944.141715] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7944.147341] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7944.151587] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7944.158005] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7944.164240] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7944.169158] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7944.174957] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7944.183777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7944.185933] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7944.191522] Call Trace: [ 7944.192463] [ 7944.196504] ? show_regs.cold.9+0x22/0x2f [ 7944.198211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7944.203481] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7944.208911] ? watchdog+0x40/0x40 [ 7944.210843] ? hrtimer_interrupt+0x128/0x2c0 [ 7944.215359] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7944.218554] ? apic_timer_interrupt+0xf/0x20 [ 7944.221331] [ 7944.224235] ? native_safe_halt+0x2e/0x50 [ 7944.227590] kvm_wait+0xcd/0xe0 [ 7944.228847] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7944.231002] do_raw_spin_lock+0x177/0x190 [ 7944.234228] _raw_spin_lock+0x21/0x30 [ 7944.235946] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7944.239308] ? do_raw_spin_unlock+0x75/0x190 [ 7944.241539] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7944.243957] ? do_wait_intr+0xf0/0xf0 [ 7944.247226] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7944.250056] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7944.253655] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7944.256799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7944.260478] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7944.262988] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7944.265445] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7944.269599] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7944.272546] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7944.278273] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7944.283950] lmv_intent_open+0x338/0x1150 [lmv] [ 7944.289690] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7944.292046] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7944.296571] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7944.299438] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7944.301570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7944.306283] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7944.308571] ? do_raw_spin_unlock+0x75/0x190 [ 7944.311410] ? _raw_spin_unlock+0x12/0x30 [ 7944.313557] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7944.317266] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7944.318737] ll_atomic_open+0x36a/0xd70 [lustre] [ 7944.320133] lookup_open+0x356/0xb40 [ 7944.328636] do_last+0x313/0xfc0 [ 7944.329588] ? nd_jump_root+0xe5/0x160 [ 7944.331375] ? path_init+0x437/0x520 [ 7944.334377] path_openat+0xf7/0x500 [ 7944.336290] ? do_raw_spin_unlock+0x75/0x190 [ 7944.338982] do_filp_open+0x99/0x140 [ 7944.340286] ? getname_flags+0x6e/0x330 [ 7944.341538] ? __check_object_size+0xff/0x256 [ 7944.343328] ? do_raw_spin_unlock+0x75/0x190 [ 7944.345593] ? _raw_spin_unlock+0x12/0x30 [ 7944.347339] do_sys_openat2+0x2b4/0x410 [ 7944.348738] do_sys_open+0x73/0xa0 [ 7944.350031] __x64_sys_openat+0x24/0x30 [ 7944.356330] do_syscall_64+0xc1/0x440 [ 7944.357781] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7944.360405] RIP: 0033:0x7f1a63983332 [ 7944.361945] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7944.369341] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7944.372821] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7944.376832] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7944.379749] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7944.383052] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7944.386272] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7947.746682] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5924s! [ 7947.750560] Showing busy workqueues and worker pools: [ 7947.753608] workqueue events: flags=0x0 [ 7947.758632] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7947.766000] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7947.774166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7947.780481] pending: sec_gc_main [ptlrpc] [ 7947.784591] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7947.794501] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7947.804547] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7947.810491] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7947.824505] workqueue events_long: flags=0x0 [ 7947.828213] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7947.831540] pending: lu_objects_flush [obdclass] [ 7947.834145] workqueue events_unbound: flags=0x2 [ 7947.835512] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7947.841508] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7947.855945] workqueue events_power_efficient: flags=0x80 [ 7947.862621] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7947.865587] pending: neigh_periodic_work, do_cache_clean [ 7947.868750] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7947.873716] pending: neigh_periodic_work, check_lifetime [ 7947.875698] workqueue mm_percpu_wq: flags=0x8 [ 7947.878527] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7947.882381] pending: vmstat_update [ 7947.883918] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7947.886430] pending: vmstat_update [ 7947.888752] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7947.893778] pending: vmstat_update [ 7947.895462] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7947.900346] pending: vmstat_update [ 7947.901483] workqueue writeback: flags=0x4a [ 7947.903727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7947.906445] pending: wb_workfn, wb_workfn [ 7947.908446] workqueue xprtiod: flags=0xa [ 7947.913488] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7947.918490] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7947.925649] workqueue ipv6_addrconf: flags=0x40008 [ 7947.928912] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7947.935650] pending: addrconf_verify_work [ 7947.937346] workqueue knbd0-recv: flags=0x1a [ 7947.939289] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7947.944644] in-flight: 647:recv_work [ 7947.951626] workqueue ptlrpc_pinger: flags=0x2 [ 7947.959529] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7947.965368] pending: ptlrpc_pinger_main [ptlrpc] [ 7947.978720] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7938s workers=2 idle: 652 [ 7952.058008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 7952.064120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7952.079352] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7952.082748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7952.085439] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7952.087770] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7952.094327] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7952.096555] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7952.099341] RDX: 000000000000118a RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7952.101882] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7952.104744] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7952.107501] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7952.109901] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7952.114235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7952.116624] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7952.120048] Call Trace: [ 7952.121060] [ 7952.121867] ? show_regs.cold.9+0x22/0x2f [ 7952.123307] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7952.125384] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7952.127543] ? watchdog+0x40/0x40 [ 7952.128885] ? hrtimer_interrupt+0x128/0x2c0 [ 7952.130415] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7952.132432] ? apic_timer_interrupt+0xf/0x20 [ 7952.133984] [ 7952.134950] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7952.137298] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7952.139224] do_raw_spin_lock+0x177/0x190 [ 7952.140634] _raw_spin_lock+0x21/0x30 [ 7952.142108] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7952.144053] obd_mod_free+0x46/0x140 [ptlrpc] [ 7952.146093] mdc_commit_open+0x91/0xb0 [mdc] [ 7952.147479] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7952.149923] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7952.152156] after_reply+0xe2c/0x1780 [ptlrpc] [ 7952.154180] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7952.156185] ? __next_timer_interrupt+0x160/0x160 [ 7952.157880] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7952.159530] ? do_wait_intr+0xf0/0xf0 [ 7952.161001] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7952.162871] kthread+0x1d1/0x200 [ 7952.164027] ? set_kthread_struct+0x70/0x70 [ 7952.165155] ret_from_fork+0x1f/0x30 [ 7956.057021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7956.059776] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7956.077059] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7956.083506] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7956.087438] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7956.089486] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7956.096938] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7956.100137] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7956.103510] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7956.106333] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7956.109086] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7956.111820] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7956.120126] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7956.123629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7956.125832] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7956.128356] Call Trace: [ 7956.129317] [ 7956.130137] ? show_regs.cold.9+0x22/0x2f [ 7956.131456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7956.133401] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7956.135179] ? watchdog+0x40/0x40 [ 7956.136273] ? hrtimer_interrupt+0x128/0x2c0 [ 7956.138139] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7956.140134] ? apic_timer_interrupt+0xf/0x20 [ 7956.141897] [ 7956.142798] ? native_safe_halt+0x2e/0x50 [ 7956.144327] kvm_wait+0xcd/0xe0 [ 7956.145642] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7956.147671] do_raw_spin_lock+0x177/0x190 [ 7956.149574] _raw_spin_lock+0x21/0x30 [ 7956.151124] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7956.153568] ? do_raw_spin_unlock+0x75/0x190 [ 7956.155377] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7956.159867] ? do_wait_intr+0xf0/0xf0 [ 7956.161298] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7956.163776] mdc_close+0x53b/0xd70 [mdc] [ 7956.165363] lmv_close+0x18d/0x490 [lmv] [ 7956.166923] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7956.169320] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7956.171487] ll_md_close+0x274/0x800 [lustre] [ 7956.173445] ll_file_release+0x128/0xc50 [lustre] [ 7956.175465] __fput+0x10d/0x440 [ 7956.176500] ____fput+0x12/0x20 [ 7956.177629] task_work_run+0xc6/0x110 [ 7956.178740] exit_to_usermode_loop+0x1dd/0x1f0 [ 7956.180456] do_syscall_64+0x3d6/0x440 [ 7956.181682] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7956.183526] RIP: 0033:0x7f1a63983cd8 [ 7956.184858] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7956.191424] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7956.194310] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7956.196709] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7956.199313] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7956.201819] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7956.204459] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7968.059323] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7968.064037] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7968.087944] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7968.093897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7968.101126] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7968.104028] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7968.114421] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7968.118912] RAX: 0000000000007118 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7968.123045] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7968.127600] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7968.131021] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7968.134410] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7968.139211] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7968.149089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7968.158015] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7968.164081] Call Trace: [ 7968.166302] [ 7968.167104] ? show_regs.cold.9+0x22/0x2f [ 7968.174582] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7968.180303] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7968.189842] ? watchdog+0x40/0x40 [ 7968.193272] ? hrtimer_interrupt+0x128/0x2c0 [ 7968.197932] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7968.203992] ? apic_timer_interrupt+0xf/0x20 [ 7968.210833] [ 7968.211870] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7968.219598] do_raw_spin_lock+0x177/0x190 [ 7968.221871] _raw_spin_lock+0x21/0x30 [ 7968.224360] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7968.230618] ? do_raw_spin_unlock+0x75/0x190 [ 7968.233510] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7968.239445] ? do_wait_intr+0xf0/0xf0 [ 7968.243471] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7968.254278] mdc_close+0x53b/0xd70 [mdc] [ 7968.258369] lmv_close+0x18d/0x490 [lmv] [ 7968.260839] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7968.265795] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7968.271546] ll_md_close+0x274/0x800 [lustre] [ 7968.278367] ll_file_release+0x128/0xc50 [lustre] [ 7968.283781] __fput+0x10d/0x440 [ 7968.285763] ____fput+0x12/0x20 [ 7968.287549] task_work_run+0xc6/0x110 [ 7968.290083] exit_to_usermode_loop+0x1dd/0x1f0 [ 7968.292459] do_syscall_64+0x3d6/0x440 [ 7968.294487] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7968.299304] RIP: 0033:0x7f1a63983cd8 [ 7968.301356] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7968.315927] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7968.326073] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7968.333989] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7968.338321] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7968.343509] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7968.350798] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7972.053026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 7972.056413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7972.097144] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7972.105557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7972.112561] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7972.115139] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7972.129585] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7972.141049] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7972.149712] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 7972.153931] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 7972.158205] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 7972.162509] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 7972.168543] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 7972.176170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7972.182363] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 7972.187956] Call Trace: [ 7972.189154] [ 7972.194041] ? show_regs.cold.9+0x22/0x2f [ 7972.195631] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7972.199224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7972.202019] ? watchdog+0x40/0x40 [ 7972.204158] ? hrtimer_interrupt+0x128/0x2c0 [ 7972.209827] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7972.216961] ? apic_timer_interrupt+0xf/0x20 [ 7972.219307] [ 7972.220899] ? native_safe_halt+0x2e/0x50 [ 7972.223144] kvm_wait+0xcd/0xe0 [ 7972.225846] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7972.229644] do_raw_spin_lock+0x177/0x190 [ 7972.231307] _raw_spin_lock+0x21/0x30 [ 7972.233702] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7972.240717] ? do_raw_spin_unlock+0x75/0x190 [ 7972.245232] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7972.251776] ? do_wait_intr+0xf0/0xf0 [ 7972.253724] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7972.261790] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7972.268546] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7972.279063] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7972.284410] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7972.287421] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7972.298076] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7972.299734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7972.305360] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7972.312125] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7972.317502] lmv_intent_open+0x338/0x1150 [lmv] [ 7972.322198] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7972.329287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7972.336190] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7972.339628] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7972.344651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7972.347715] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7972.350555] ? do_raw_spin_unlock+0x75/0x190 [ 7972.353671] ? _raw_spin_unlock+0x12/0x30 [ 7972.356063] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7972.358578] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7972.361421] ll_atomic_open+0x36a/0xd70 [lustre] [ 7972.364840] lookup_open+0x356/0xb40 [ 7972.366677] do_last+0x313/0xfc0 [ 7972.369251] ? nd_jump_root+0xe5/0x160 [ 7972.373544] ? path_init+0x437/0x520 [ 7972.378259] path_openat+0xf7/0x500 [ 7972.381620] ? do_raw_spin_unlock+0x75/0x190 [ 7972.383241] do_filp_open+0x99/0x140 [ 7972.385960] ? getname_flags+0x6e/0x330 [ 7972.390150] ? __check_object_size+0xff/0x256 [ 7972.395254] ? do_raw_spin_unlock+0x75/0x190 [ 7972.397535] ? _raw_spin_unlock+0x12/0x30 [ 7972.402259] do_sys_openat2+0x2b4/0x410 [ 7972.404500] do_sys_open+0x73/0xa0 [ 7972.406266] __x64_sys_openat+0x24/0x30 [ 7972.408361] do_syscall_64+0xc1/0x440 [ 7972.412483] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7972.415704] RIP: 0033:0x7f1a63983332 [ 7972.417673] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7972.432343] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7972.442520] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 7972.449951] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 7972.456292] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 7972.461883] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 7972.466189] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 7978.464587] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5955s! [ 7978.468241] Showing busy workqueues and worker pools: [ 7978.471767] workqueue events: flags=0x0 [ 7978.473383] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7978.480910] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 7978.484165] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7978.487097] pending: sec_gc_main [ptlrpc] [ 7978.489787] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7978.492936] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 7978.498828] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7978.501881] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 7978.506496] workqueue events_long: flags=0x0 [ 7978.508378] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7978.511554] pending: lu_objects_flush [obdclass] [ 7978.513662] workqueue events_unbound: flags=0x2 [ 7978.517392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7978.519351] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7978.526151] workqueue events_power_efficient: flags=0x80 [ 7978.533416] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7978.539287] pending: neigh_periodic_work, do_cache_clean [ 7978.542307] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7978.547898] pending: neigh_periodic_work, check_lifetime [ 7978.555785] workqueue mm_percpu_wq: flags=0x8 [ 7978.562636] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7978.571530] pending: vmstat_update [ 7978.572963] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7978.580961] pending: vmstat_update [ 7978.582857] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7978.585920] pending: vmstat_update [ 7978.588403] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7978.591760] pending: vmstat_update [ 7978.593624] workqueue writeback: flags=0x4a [ 7978.600108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7978.602847] pending: wb_workfn, wb_workfn [ 7978.604657] workqueue xprtiod: flags=0xa [ 7978.606721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7978.608810] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 7978.612974] workqueue ipv6_addrconf: flags=0x40008 [ 7978.616850] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7978.619690] pending: addrconf_verify_work [ 7978.621720] workqueue knbd0-recv: flags=0x1a [ 7978.623516] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7978.626934] in-flight: 647:recv_work [ 7978.630215] workqueue ptlrpc_pinger: flags=0x2 [ 7978.632083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7978.635124] pending: ptlrpc_pinger_main [ptlrpc] [ 7978.637110] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7968s workers=2 idle: 652 [ 7980.058018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 7980.064354] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7980.095013] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7980.105291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7980.112980] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7980.117301] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7980.131719] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7980.136908] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 7980.139849] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 7980.145458] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 7980.149200] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 7980.156214] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7980.161793] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 7980.168637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7980.173564] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 7980.184380] Call Trace: [ 7980.185086] [ 7980.185637] ? show_regs.cold.9+0x22/0x2f [ 7980.187232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7980.189955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7980.194932] ? watchdog+0x40/0x40 [ 7980.197037] ? hrtimer_interrupt+0x128/0x2c0 [ 7980.198258] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7980.200882] ? apic_timer_interrupt+0xf/0x20 [ 7980.202125] [ 7980.203944] ? native_safe_halt+0x2e/0x50 [ 7980.208827] kvm_wait+0xcd/0xe0 [ 7980.215150] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7980.218867] do_raw_spin_lock+0x177/0x190 [ 7980.220189] _raw_spin_lock+0x21/0x30 [ 7980.228035] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7980.238337] obd_mod_free+0x46/0x140 [ptlrpc] [ 7980.243065] mdc_commit_open+0x91/0xb0 [mdc] [ 7980.244449] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7980.248713] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7980.254439] after_reply+0xe2c/0x1780 [ptlrpc] [ 7980.257181] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7980.259688] ? __next_timer_interrupt+0x160/0x160 [ 7980.261911] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7980.264495] ? do_wait_intr+0xf0/0xf0 [ 7980.266695] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7980.269708] kthread+0x1d1/0x200 [ 7980.271725] ? set_kthread_struct+0x70/0x70 [ 7980.276478] ret_from_fork+0x1f/0x30 [ 7984.057034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 7984.059985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7984.080651] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7984.084354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7984.087679] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7984.090225] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7984.097280] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7984.099998] RAX: 0000000000005c93 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7984.102650] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 7984.104942] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 7984.107584] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 7984.110035] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 7984.112889] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 7984.115701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7984.117492] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 7984.120003] Call Trace: [ 7984.120799] [ 7984.121445] ? show_regs.cold.9+0x22/0x2f [ 7984.123441] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7984.125622] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7984.127535] ? watchdog+0x40/0x40 [ 7984.129377] ? hrtimer_interrupt+0x128/0x2c0 [ 7984.131279] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7984.134012] ? apic_timer_interrupt+0xf/0x20 [ 7984.135435] [ 7984.136119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7984.138345] do_raw_spin_lock+0x177/0x190 [ 7984.140532] _raw_spin_lock+0x21/0x30 [ 7984.142633] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7984.144820] ? do_raw_spin_unlock+0x75/0x190 [ 7984.146182] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7984.148253] ? do_wait_intr+0xf0/0xf0 [ 7984.149292] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7984.151348] mdc_close+0x53b/0xd70 [mdc] [ 7984.152848] lmv_close+0x18d/0x490 [lmv] [ 7984.154410] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7984.156584] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7984.158632] ll_md_close+0x274/0x800 [lustre] [ 7984.159999] ll_file_release+0x128/0xc50 [lustre] [ 7984.161960] __fput+0x10d/0x440 [ 7984.162991] ____fput+0x12/0x20 [ 7984.164182] task_work_run+0xc6/0x110 [ 7984.165405] exit_to_usermode_loop+0x1dd/0x1f0 [ 7984.167303] do_syscall_64+0x3d6/0x440 [ 7984.168618] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7984.170874] RIP: 0033:0x7f1a63983cd8 [ 7984.172116] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7984.178795] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7984.181932] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 7984.184398] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7984.187055] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 7984.189700] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 7984.192585] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7996.060141] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 7996.068822] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7996.106589] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7996.120941] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7996.125813] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.130295] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7996.161208] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7996.169401] RAX: 000000000000440e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 7996.174294] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 7996.178426] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7996.184327] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 7996.191625] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 7996.195670] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 7996.200510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7996.203269] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 7996.207251] Call Trace: [ 7996.208492] [ 7996.209652] ? show_regs.cold.9+0x22/0x2f [ 7996.215089] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7996.221454] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7996.230020] ? watchdog+0x40/0x40 [ 7996.234069] ? hrtimer_interrupt+0x128/0x2c0 [ 7996.240156] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7996.246507] ? apic_timer_interrupt+0xf/0x20 [ 7996.249588] [ 7996.254334] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.262336] do_raw_spin_lock+0x177/0x190 [ 7996.267649] _raw_spin_lock+0x21/0x30 [ 7996.274489] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7996.278521] ? do_raw_spin_unlock+0x75/0x190 [ 7996.281473] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7996.284181] ? do_wait_intr+0xf0/0xf0 [ 7996.286425] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7996.293631] mdc_close+0x53b/0xd70 [mdc] [ 7996.296532] lmv_close+0x18d/0x490 [lmv] [ 7996.298274] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7996.310051] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7996.312033] ll_md_close+0x274/0x800 [lustre] [ 7996.314124] ll_file_release+0x128/0xc50 [lustre] [ 7996.316918] __fput+0x10d/0x440 [ 7996.318975] ____fput+0x12/0x20 [ 7996.321032] task_work_run+0xc6/0x110 [ 7996.322522] exit_to_usermode_loop+0x1dd/0x1f0 [ 7996.324546] do_syscall_64+0x3d6/0x440 [ 7996.326302] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7996.328947] RIP: 0033:0x7f1a63983cd8 [ 7996.330480] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7996.339301] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7996.342587] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 7996.345975] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 7996.349308] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 7996.352836] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 7996.356280] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8004.053183] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8004.057372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8004.086093] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8004.092886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8004.098141] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8004.101645] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8004.111427] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8004.116748] RAX: 0000000000007a0d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8004.119695] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8004.123036] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8004.126273] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8004.128895] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8004.131578] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8004.134788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8004.138734] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8004.142819] Call Trace: [ 8004.144197] [ 8004.146432] ? show_regs.cold.9+0x22/0x2f [ 8004.149390] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8004.153358] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8004.156121] ? watchdog+0x40/0x40 [ 8004.157410] ? hrtimer_interrupt+0x128/0x2c0 [ 8004.160026] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8004.162640] ? apic_timer_interrupt+0xf/0x20 [ 8004.164601] [ 8004.165714] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8004.168516] do_raw_spin_lock+0x177/0x190 [ 8004.170235] _raw_spin_lock+0x21/0x30 [ 8004.172342] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8004.175223] ? do_raw_spin_unlock+0x75/0x190 [ 8004.177743] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8004.180707] ? do_wait_intr+0xf0/0xf0 [ 8004.183096] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8004.186264] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8004.189590] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8004.193934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8004.196815] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8004.200132] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8004.203515] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8004.205940] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8004.209208] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8004.212961] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8004.215416] lmv_intent_open+0x338/0x1150 [lmv] [ 8004.217351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8004.219651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8004.221796] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8004.223773] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8004.226120] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8004.228184] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8004.230627] ? do_raw_spin_unlock+0x75/0x190 [ 8004.232647] ? _raw_spin_unlock+0x12/0x30 [ 8004.234473] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8004.236969] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8004.238992] ll_atomic_open+0x36a/0xd70 [lustre] [ 8004.241101] lookup_open+0x356/0xb40 [ 8004.242426] do_last+0x313/0xfc0 [ 8004.243634] ? nd_jump_root+0xe5/0x160 [ 8004.245459] ? path_init+0x437/0x520 [ 8004.247257] path_openat+0xf7/0x500 [ 8004.248704] ? do_raw_spin_unlock+0x75/0x190 [ 8004.250645] do_filp_open+0x99/0x140 [ 8004.252066] ? getname_flags+0x6e/0x330 [ 8004.254045] ? __check_object_size+0xff/0x256 [ 8004.255672] ? do_raw_spin_unlock+0x75/0x190 [ 8004.257248] ? _raw_spin_unlock+0x12/0x30 [ 8004.259373] do_sys_openat2+0x2b4/0x410 [ 8004.260964] do_sys_open+0x73/0xa0 [ 8004.262802] __x64_sys_openat+0x24/0x30 [ 8004.264319] do_syscall_64+0xc1/0x440 [ 8004.265881] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8004.268223] RIP: 0033:0x7f1a63983332 [ 8004.269842] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8004.277707] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8004.280877] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8004.283450] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8004.287018] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8004.290959] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8004.294984] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8008.059569] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8008.067691] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8008.130469] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8008.139393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8008.146928] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8008.157849] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8008.186850] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8008.195604] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8008.207809] RDX: 000000000000679f RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8008.216666] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8008.224900] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8008.236312] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8008.246774] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8008.261219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8008.271016] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8008.285211] Call Trace: [ 8008.291712] [ 8008.293706] ? show_regs.cold.9+0x22/0x2f [ 8008.300716] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8008.307794] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8008.316400] ? watchdog+0x40/0x40 [ 8008.326416] ? hrtimer_interrupt+0x128/0x2c0 [ 8008.333737] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8008.342145] ? apic_timer_interrupt+0xf/0x20 [ 8008.345288] [ 8008.346435] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8008.353165] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8008.355519] do_raw_spin_lock+0x177/0x190 [ 8008.357910] _raw_spin_lock+0x21/0x30 [ 8008.362063] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8008.367073] obd_mod_free+0x46/0x140 [ptlrpc] [ 8008.373333] mdc_commit_open+0x91/0xb0 [mdc] [ 8008.381318] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8008.383048] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8008.394913] after_reply+0xe2c/0x1780 [ptlrpc] [ 8008.404335] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8008.411703] ? __next_timer_interrupt+0x160/0x160 [ 8008.414025] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8008.416162] ? do_wait_intr+0xf0/0xf0 [ 8008.417863] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8008.424237] kthread+0x1d1/0x200 [ 8008.426265] ? set_kthread_struct+0x70/0x70 [ 8008.428381] ret_from_fork+0x1f/0x30 [ 8008.673122] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5985s! [ 8008.681635] Showing busy workqueues and worker pools: [ 8008.689979] workqueue events: flags=0x0 [ 8008.697914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8008.704060] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8008.717736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8008.724088] pending: sec_gc_main [ptlrpc] [ 8008.735816] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8008.738425] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8008.755831] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8008.778612] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8008.790777] workqueue events_long: flags=0x0 [ 8008.799603] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8008.810861] pending: lu_objects_flush [obdclass] [ 8008.813954] workqueue events_unbound: flags=0x2 [ 8008.828647] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8008.834601] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8008.846330] workqueue events_power_efficient: flags=0x80 [ 8008.855797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8008.867360] pending: neigh_periodic_work, do_cache_clean [ 8008.878814] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8008.886057] pending: neigh_periodic_work, check_lifetime [ 8008.893422] workqueue mm_percpu_wq: flags=0x8 [ 8008.904519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8008.916748] pending: vmstat_update [ 8008.924575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8008.930166] pending: vmstat_update [ 8008.931152] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8008.946447] pending: vmstat_update [ 8008.954602] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8008.966502] pending: vmstat_update [ 8008.970009] workqueue writeback: flags=0x4a [ 8008.977379] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8008.982037] pending: wb_workfn, wb_workfn [ 8008.984809] workqueue xprtiod: flags=0xa [ 8008.987022] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8008.999439] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8009.011141] workqueue ipv6_addrconf: flags=0x40008 [ 8009.022715] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8009.030033] pending: addrconf_verify_work [ 8009.036924] workqueue knbd0-recv: flags=0x1a [ 8009.043458] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8009.048118] in-flight: 647:recv_work [ 8009.051584] workqueue ptlrpc_pinger: flags=0x2 [ 8009.054985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8009.060775] pending: ptlrpc_pinger_main [ptlrpc] [ 8009.067110] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7999s workers=2 idle: 652 [ 8012.058257] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8012.066298] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8012.119771] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8012.141673] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8012.159616] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8012.166478] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8012.206785] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8012.218835] RAX: 0000000000007288 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8012.226056] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8012.231987] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8012.239480] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8012.249570] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8012.263532] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8012.276611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8012.285767] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8012.292155] Call Trace: [ 8012.293696] [ 8012.294720] ? show_regs.cold.9+0x22/0x2f [ 8012.297559] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8012.305068] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8012.310006] ? watchdog+0x40/0x40 [ 8012.311178] ? hrtimer_interrupt+0x128/0x2c0 [ 8012.319801] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8012.329679] ? apic_timer_interrupt+0xf/0x20 [ 8012.334538] [ 8012.336222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8012.346668] do_raw_spin_lock+0x177/0x190 [ 8012.351746] _raw_spin_lock+0x21/0x30 [ 8012.359635] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8012.374455] ? do_raw_spin_unlock+0x75/0x190 [ 8012.376162] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8012.383298] ? do_wait_intr+0xf0/0xf0 [ 8012.388741] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8012.395635] mdc_close+0x53b/0xd70 [mdc] [ 8012.400073] lmv_close+0x18d/0x490 [lmv] [ 8012.405564] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8012.412626] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8012.416248] ll_md_close+0x274/0x800 [lustre] [ 8012.421546] ll_file_release+0x128/0xc50 [lustre] [ 8012.429531] __fput+0x10d/0x440 [ 8012.435272] ____fput+0x12/0x20 [ 8012.444414] task_work_run+0xc6/0x110 [ 8012.447303] exit_to_usermode_loop+0x1dd/0x1f0 [ 8012.449645] do_syscall_64+0x3d6/0x440 [ 8012.452297] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8012.455497] RIP: 0033:0x7f1a63983cd8 [ 8012.458605] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8012.482805] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8012.493661] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8012.505343] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8012.520550] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8012.528081] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8012.535642] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8022.561520] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 8022.569444] rcu: 3-...!: (3778846 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 8022.577121] (detected by 1, t=6000155 jiffies, g=269633, q=2202) [ 8022.582779] Sending NMI from CPU 1 to CPUs 3: [ 8022.603767] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 6.008 msecs [ 8022.603898] NMI backtrace for cpu 3 [ 8022.603902] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8022.603904] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8022.603906] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8022.603908] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8022.603910] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 8022.603913] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8022.603914] RDX: 0000000000000c32 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8022.603916] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8022.603917] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8022.603919] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8022.603920] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8022.603921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8022.603922] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8022.603924] Call Trace: [ 8022.603925] [ 8022.603926] ? show_regs.cold.9+0x22/0x2f [ 8022.603927] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8022.603928] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8022.603929] ? nmi_handle+0xbd/0x210 [ 8022.603930] ? default_do_nmi+0x4b/0x1b0 [ 8022.603931] ? do_nmi+0x280/0x380 [ 8022.603932] ? end_repeat_nmi+0x16/0x1f [ 8022.603933] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8022.603935] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8022.603936] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8022.603937] [ 8022.603938] do_raw_spin_lock+0x177/0x190 [ 8022.603940] _raw_spin_lock+0x21/0x30 [ 8022.603941] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8022.603942] obd_mod_free+0x46/0x140 [ptlrpc] [ 8022.603943] mdc_commit_open+0x91/0xb0 [mdc] [ 8022.603944] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8022.603945] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8022.603947] after_reply+0xe2c/0x1780 [ptlrpc] [ 8022.603948] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8022.603950] ? __next_timer_interrupt+0x160/0x160 [ 8022.603951] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8022.603952] ? do_wait_intr+0xf0/0xf0 [ 8022.603953] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8022.603954] kthread+0x1d1/0x200 [ 8022.603955] ? set_kthread_struct+0x70/0x70 [ 8022.603957] ret_from_fork+0x1f/0x30 [ 8022.603962] rcu: rcu_sched kthread starved for 5999801 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 8022.904929] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8022.914543] rcu: RCU grace-period kthread stack dump: [ 8022.923872] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8022.930511] Call Trace: [ 8022.935253] __schedule+0x351/0xcb0 [ 8022.939570] schedule+0xc0/0x180 [ 8022.941381] schedule_timeout+0xb4/0x190 [ 8022.945092] ? do_raw_spin_unlock+0x75/0x190 [ 8022.950352] ? __next_timer_interrupt+0x160/0x160 [ 8022.955745] ? prepare_to_swait_event+0xf3/0x140 [ 8022.960925] rcu_gp_kthread+0x8dd/0x15b0 [ 8022.964409] ? force_qs_rnp+0x2c0/0x2c0 [ 8022.968335] kthread+0x1d1/0x200 [ 8022.969624] ? set_kthread_struct+0x70/0x70 [ 8022.973163] ret_from_fork+0x1f/0x30 [ 8022.975016] rcu: Stack dump where RCU GP kthread last ran: [ 8022.978083] Sending NMI from CPU 1 to CPUs 3: [ 8022.986460] NMI backtrace for cpu 3 [ 8022.986468] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8022.986470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8022.986472] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8022.986473] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8022.986475] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 8022.986477] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8022.986479] RDX: 0000000000006631 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8022.986480] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8022.986482] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8022.986483] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8022.986485] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8022.986486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8022.986487] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8022.986488] Call Trace: [ 8022.986489] [ 8022.986490] ? show_regs.cold.9+0x22/0x2f [ 8022.986491] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8022.986492] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8022.986494] ? nmi_handle+0xbd/0x210 [ 8022.986494] ? default_do_nmi+0x4b/0x1b0 [ 8022.986495] ? do_nmi+0x280/0x380 [ 8022.986496] ? end_repeat_nmi+0x16/0x1f [ 8022.986497] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8022.986499] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8022.986500] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8022.986501] [ 8022.986502] do_raw_spin_lock+0x177/0x190 [ 8022.986503] _raw_spin_lock+0x21/0x30 [ 8022.986504] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8022.986505] obd_mod_free+0x46/0x140 [ptlrpc] [ 8022.986507] mdc_commit_open+0x91/0xb0 [mdc] [ 8022.986508] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8022.986509] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8022.986511] after_reply+0xe2c/0x1780 [ptlrpc] [ 8022.986512] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8022.986513] ? __next_timer_interrupt+0x160/0x160 [ 8022.986514] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8022.986515] ? do_wait_intr+0xf0/0xf0 [ 8022.986516] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8022.986518] kthread+0x1d1/0x200 [ 8022.986518] ? set_kthread_struct+0x70/0x70 [ 8022.986519] ret_from_fork+0x1f/0x30 [ 8036.053030] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 8036.057547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8036.082954] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8036.087368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8036.092290] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8036.094587] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8036.104231] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8036.108080] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8036.111783] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8036.115959] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8036.119609] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8036.124267] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8036.128056] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8036.132277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8036.135779] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8036.139421] Call Trace: [ 8036.140564] [ 8036.141580] ? show_regs.cold.9+0x22/0x2f [ 8036.143565] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8036.145962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8036.148234] ? watchdog+0x40/0x40 [ 8036.149924] ? hrtimer_interrupt+0x128/0x2c0 [ 8036.152079] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8036.154394] ? apic_timer_interrupt+0xf/0x20 [ 8036.156936] [ 8036.158168] ? native_safe_halt+0x2e/0x50 [ 8036.160310] kvm_wait+0xcd/0xe0 [ 8036.162012] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8036.164073] do_raw_spin_lock+0x177/0x190 [ 8036.166169] _raw_spin_lock+0x21/0x30 [ 8036.167800] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8036.170343] ? do_raw_spin_unlock+0x75/0x190 [ 8036.172082] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8036.174402] ? do_wait_intr+0xf0/0xf0 [ 8036.176201] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8036.178620] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8036.181249] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8036.184322] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8036.187088] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8036.189528] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8036.191622] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8036.193646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8036.202336] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8036.207277] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8036.214232] lmv_intent_open+0x338/0x1150 [lmv] [ 8036.217637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8036.220557] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8036.223535] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8036.226138] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8036.228446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8036.230923] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8036.233091] ? do_raw_spin_unlock+0x75/0x190 [ 8036.235376] ? _raw_spin_unlock+0x12/0x30 [ 8036.238527] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8036.241239] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8036.243452] ll_atomic_open+0x36a/0xd70 [lustre] [ 8036.245622] lookup_open+0x356/0xb40 [ 8036.247079] do_last+0x313/0xfc0 [ 8036.248353] ? nd_jump_root+0xe5/0x160 [ 8036.249881] ? path_init+0x437/0x520 [ 8036.251518] path_openat+0xf7/0x500 [ 8036.253433] ? do_raw_spin_unlock+0x75/0x190 [ 8036.255408] do_filp_open+0x99/0x140 [ 8036.257299] ? getname_flags+0x6e/0x330 [ 8036.259096] ? __check_object_size+0xff/0x256 [ 8036.261121] ? do_raw_spin_unlock+0x75/0x190 [ 8036.263085] ? _raw_spin_unlock+0x12/0x30 [ 8036.264843] do_sys_openat2+0x2b4/0x410 [ 8036.267298] do_sys_open+0x73/0xa0 [ 8036.269160] __x64_sys_openat+0x24/0x30 [ 8036.270898] do_syscall_64+0xc1/0x440 [ 8036.272451] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8036.274852] RIP: 0033:0x7f1a63983332 [ 8036.277370] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8036.294166] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8036.299403] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8036.306241] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8036.311337] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8036.315412] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8036.322608] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8039.395791] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6016s! [ 8039.400489] Showing busy workqueues and worker pools: [ 8039.404703] workqueue events: flags=0x0 [ 8039.406157] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8039.414888] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8039.425717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8039.431611] pending: sec_gc_main [ptlrpc] [ 8039.434341] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8039.437822] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8039.442562] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8039.444924] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8039.448754] workqueue events_long: flags=0x0 [ 8039.450584] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8039.453237] pending: lu_objects_flush [obdclass] [ 8039.455502] workqueue events_unbound: flags=0x2 [ 8039.457238] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8039.459567] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8039.464051] workqueue events_power_efficient: flags=0x80 [ 8039.466346] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8039.469088] pending: neigh_periodic_work, do_cache_clean [ 8039.473824] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8039.476828] pending: neigh_periodic_work, check_lifetime [ 8039.479651] workqueue mm_percpu_wq: flags=0x8 [ 8039.481922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8039.486503] pending: vmstat_update [ 8039.488458] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8039.495029] pending: vmstat_update [ 8039.497697] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8039.503156] pending: vmstat_update [ 8039.505274] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8039.507935] pending: vmstat_update [ 8039.509585] workqueue writeback: flags=0x4a [ 8039.511432] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8039.513794] pending: wb_workfn, wb_workfn [ 8039.515330] workqueue xprtiod: flags=0xa [ 8039.516699] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8039.519315] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8039.522647] workqueue ipv6_addrconf: flags=0x40008 [ 8039.524667] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8039.527491] pending: addrconf_verify_work [ 8039.529377] workqueue knbd0-recv: flags=0x1a [ 8039.531081] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8039.534647] in-flight: 647:recv_work [ 8039.536644] workqueue ptlrpc_pinger: flags=0x2 [ 8039.538670] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8039.541644] pending: ptlrpc_pinger_main [ptlrpc] [ 8039.544481] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8029s workers=2 idle: 652 [ 8040.057053] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8040.059695] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8040.097389] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8040.105840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8040.112269] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8040.114142] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8040.121361] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8040.124518] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8040.127459] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8040.130488] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8040.133413] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8040.136302] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8040.139027] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8040.142262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8040.144970] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8040.148246] Call Trace: [ 8040.149189] [ 8040.150055] ? show_regs.cold.9+0x22/0x2f [ 8040.152187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8040.154015] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8040.155753] ? watchdog+0x40/0x40 [ 8040.157353] ? hrtimer_interrupt+0x128/0x2c0 [ 8040.159138] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8040.160857] ? apic_timer_interrupt+0xf/0x20 [ 8040.162348] [ 8040.163258] ? native_safe_halt+0x2e/0x50 [ 8040.164660] kvm_wait+0xcd/0xe0 [ 8040.165898] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8040.167892] do_raw_spin_lock+0x177/0x190 [ 8040.169512] _raw_spin_lock+0x21/0x30 [ 8040.171043] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8040.173478] ? do_raw_spin_unlock+0x75/0x190 [ 8040.175228] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8040.178075] ? do_wait_intr+0xf0/0xf0 [ 8040.179348] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8040.182237] mdc_close+0x53b/0xd70 [mdc] [ 8040.183832] lmv_close+0x18d/0x490 [lmv] [ 8040.185842] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8040.188241] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8040.190241] ll_md_close+0x274/0x800 [lustre] [ 8040.192048] ll_file_release+0x128/0xc50 [lustre] [ 8040.194046] __fput+0x10d/0x440 [ 8040.195144] ____fput+0x12/0x20 [ 8040.196263] task_work_run+0xc6/0x110 [ 8040.198007] exit_to_usermode_loop+0x1dd/0x1f0 [ 8040.199583] do_syscall_64+0x3d6/0x440 [ 8040.201127] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8040.203128] RIP: 0033:0x7f1a63983cd8 [ 8040.204581] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8040.212183] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8040.214974] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8040.218137] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8040.220684] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8040.223132] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8040.225611] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8048.057035] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 8048.058254] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8048.061870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8048.061953] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8048.061961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8048.061964] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8048.061985] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8048.061989] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8048.061994] RAX: 0000000000001dc2 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8048.061997] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8048.061999] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8048.062002] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8048.062004] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8048.062008] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8048.062011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8048.062013] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8048.062025] Call Trace: [ 8048.062030] [ 8048.062035] ? show_regs.cold.9+0x22/0x2f [ 8048.062044] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8048.062056] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8048.062061] ? watchdog+0x40/0x40 [ 8048.062065] ? hrtimer_interrupt+0x128/0x2c0 [ 8048.062069] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8048.062076] ? apic_timer_interrupt+0xf/0x20 [ 8048.062084] [ 8048.062087] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8048.062094] do_raw_spin_lock+0x177/0x190 [ 8048.062099] _raw_spin_lock+0x21/0x30 [ 8048.062112] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8048.062436] ? do_raw_spin_unlock+0x75/0x190 [ 8048.062447] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8048.062679] ? do_wait_intr+0xf0/0xf0 [ 8048.062693] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8048.069463] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8048.069566] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8048.069573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8048.069576] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8048.069589] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8048.069594] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8048.069599] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8048.069601] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8048.069604] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8048.069606] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8048.069609] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8048.069612] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8048.069615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8048.069618] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8048.069626] Call Trace: [ 8048.069631] [ 8048.069637] ? show_regs.cold.9+0x22/0x2f [ 8048.069646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8048.069653] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8048.069671] ? watchdog+0x40/0x40 [ 8048.069675] ? hrtimer_interrupt+0x128/0x2c0 [ 8048.069680] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8048.069686] ? apic_timer_interrupt+0xf/0x20 [ 8048.069691] [ 8048.069694] ? native_safe_halt+0x2e/0x50 [ 8048.069698] kvm_wait+0xcd/0xe0 [ 8048.069706] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8048.069713] do_raw_spin_lock+0x177/0x190 [ 8048.069718] _raw_spin_lock+0x21/0x30 [ 8048.069723] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8048.123152] mdc_close+0x53b/0xd70 [mdc] [ 8048.123360] lmv_close+0x18d/0x490 [lmv] [ 8048.123397] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8048.123538] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8048.123638] ll_md_close+0x274/0x800 [lustre] [ 8048.124121] ll_file_release+0x128/0xc50 [lustre] [ 8048.124212] __fput+0x10d/0x440 [ 8048.124298] ____fput+0x12/0x20 [ 8048.124304] task_work_run+0xc6/0x110 [ 8048.124311] exit_to_usermode_loop+0x1dd/0x1f0 [ 8048.124319] do_syscall_64+0x3d6/0x440 [ 8048.124330] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8048.124338] RIP: 0033:0x7f1a63983cd8 [ 8048.124345] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8048.124349] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8048.124358] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8048.124361] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8048.124363] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8048.124366] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8048.124368] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8048.450977] obd_mod_free+0x46/0x140 [ptlrpc] [ 8048.454561] mdc_commit_open+0x91/0xb0 [mdc] [ 8048.460076] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8048.464260] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8048.469392] after_reply+0xe2c/0x1780 [ptlrpc] [ 8048.475193] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8048.479841] ? __next_timer_interrupt+0x160/0x160 [ 8048.487519] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8048.493406] ? do_wait_intr+0xf0/0xf0 [ 8048.497348] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8048.500272] kthread+0x1d1/0x200 [ 8048.503263] ? set_kthread_struct+0x70/0x70 [ 8048.505628] ret_from_fork+0x1f/0x30 [ 8064.054538] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 8064.058312] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8064.084505] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8064.089874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8064.093885] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8064.096603] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8064.106278] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8064.109764] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8064.113227] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8064.116262] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8064.118539] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8064.121842] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8064.125639] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8064.130145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8064.133024] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8064.136789] Call Trace: [ 8064.137802] [ 8064.140259] ? show_regs.cold.9+0x22/0x2f [ 8064.142623] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8064.144923] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8064.146998] ? watchdog+0x40/0x40 [ 8064.153015] ? hrtimer_interrupt+0x128/0x2c0 [ 8064.155330] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8064.157764] ? apic_timer_interrupt+0xf/0x20 [ 8064.160328] [ 8064.161511] ? native_safe_halt+0x2e/0x50 [ 8064.167915] kvm_wait+0xcd/0xe0 [ 8064.172807] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8064.177670] do_raw_spin_lock+0x177/0x190 [ 8064.193137] _raw_spin_lock+0x21/0x30 [ 8064.196543] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8064.210613] ? do_raw_spin_unlock+0x75/0x190 [ 8064.218837] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8064.230383] ? do_wait_intr+0xf0/0xf0 [ 8064.235479] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8064.241402] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8064.244193] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8064.247397] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8064.251014] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8064.253317] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8064.255574] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8064.258115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8064.262867] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8064.265934] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8064.277681] lmv_intent_open+0x338/0x1150 [lmv] [ 8064.286335] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8064.291627] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8064.298276] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8064.302485] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8064.304639] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8064.314988] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8064.317258] ? do_raw_spin_unlock+0x75/0x190 [ 8064.319599] ? _raw_spin_unlock+0x12/0x30 [ 8064.325099] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8064.330549] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8064.332556] ll_atomic_open+0x36a/0xd70 [lustre] [ 8064.334639] lookup_open+0x356/0xb40 [ 8064.337287] do_last+0x313/0xfc0 [ 8064.339512] ? nd_jump_root+0xe5/0x160 [ 8064.343582] ? path_init+0x437/0x520 [ 8064.344612] path_openat+0xf7/0x500 [ 8064.345611] ? do_raw_spin_unlock+0x75/0x190 [ 8064.355671] do_filp_open+0x99/0x140 [ 8064.359655] ? getname_flags+0x6e/0x330 [ 8064.362324] ? __check_object_size+0xff/0x256 [ 8064.366201] ? do_raw_spin_unlock+0x75/0x190 [ 8064.369642] ? _raw_spin_unlock+0x12/0x30 [ 8064.373566] do_sys_openat2+0x2b4/0x410 [ 8064.377236] do_sys_open+0x73/0xa0 [ 8064.378913] __x64_sys_openat+0x24/0x30 [ 8064.380602] do_syscall_64+0xc1/0x440 [ 8064.382464] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8064.384396] RIP: 0033:0x7f1a63983332 [ 8064.385761] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8064.398962] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8064.402284] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8064.406637] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8064.410626] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8064.414686] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8064.419366] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8068.061058] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8068.076528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8068.150486] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8068.156732] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8068.171613] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8068.179594] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8068.209162] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8068.213328] RAX: 0000000000003387 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8068.222766] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8068.233265] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8068.241796] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8068.250354] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8068.257426] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8068.274683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8068.287587] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8068.301182] Call Trace: [ 8068.302814] [ 8068.306490] ? show_regs.cold.9+0x22/0x2f [ 8068.314795] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8068.318763] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8068.321008] ? watchdog+0x40/0x40 [ 8068.323500] ? hrtimer_interrupt+0x128/0x2c0 [ 8068.327782] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8068.331866] ? apic_timer_interrupt+0xf/0x20 [ 8068.334151] [ 8068.335736] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8068.339122] do_raw_spin_lock+0x177/0x190 [ 8068.341858] _raw_spin_lock+0x21/0x30 [ 8068.344204] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8068.347618] ? do_raw_spin_unlock+0x75/0x190 [ 8068.351513] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8068.360600] ? do_wait_intr+0xf0/0xf0 [ 8068.367796] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8068.372248] mdc_close+0x53b/0xd70 [mdc] [ 8068.374624] lmv_close+0x18d/0x490 [lmv] [ 8068.377605] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8068.382284] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8068.385252] ll_md_close+0x274/0x800 [lustre] [ 8068.388124] ll_file_release+0x128/0xc50 [lustre] [ 8068.397598] __fput+0x10d/0x440 [ 8068.398792] ____fput+0x12/0x20 [ 8068.400019] task_work_run+0xc6/0x110 [ 8068.406623] exit_to_usermode_loop+0x1dd/0x1f0 [ 8068.412161] do_syscall_64+0x3d6/0x440 [ 8068.414626] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8068.421500] RIP: 0033:0x7f1a63983cd8 [ 8068.426423] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8068.456078] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8068.467506] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8068.474246] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8068.489492] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8068.497421] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8068.510596] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8069.610145] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6046s! [ 8069.625493] Showing busy workqueues and worker pools: [ 8069.633579] workqueue events: flags=0x0 [ 8069.638473] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8069.646847] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8069.653393] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8069.660128] pending: sec_gc_main [ptlrpc] [ 8069.665499] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8069.675419] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8069.685104] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8069.692828] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8069.708399] workqueue events_long: flags=0x0 [ 8069.711328] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8069.720617] pending: lu_objects_flush [obdclass] [ 8069.726342] workqueue events_unbound: flags=0x2 [ 8069.729804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8069.733519] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8069.739331] workqueue events_power_efficient: flags=0x80 [ 8069.742217] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8069.747359] pending: neigh_periodic_work, do_cache_clean [ 8069.750505] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8069.758936] pending: neigh_periodic_work, check_lifetime [ 8069.764056] workqueue mm_percpu_wq: flags=0x8 [ 8069.767574] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8069.782716] pending: vmstat_update [ 8069.787195] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8069.794562] pending: vmstat_update [ 8069.802230] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8069.813435] pending: vmstat_update [ 8069.816040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8069.833509] pending: vmstat_update [ 8069.841515] workqueue writeback: flags=0x4a [ 8069.847484] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8069.856553] pending: wb_workfn, wb_workfn [ 8069.861327] workqueue xprtiod: flags=0xa [ 8069.869765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8069.876346] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8069.884290] workqueue ipv6_addrconf: flags=0x40008 [ 8069.893389] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8069.904580] pending: addrconf_verify_work [ 8069.909134] workqueue knbd0-recv: flags=0x1a [ 8069.913277] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8069.925158] in-flight: 647:recv_work [ 8069.930708] workqueue ptlrpc_pinger: flags=0x2 [ 8069.939246] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8069.946551] pending: ptlrpc_pinger_main [ptlrpc] [ 8069.960529] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8059s workers=2 idle: 652 [ 8076.059992] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8076.066264] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8076.072924] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8076.074088] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8076.074101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8076.074104] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8076.074121] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8076.074125] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8076.074129] RAX: 0000000000007907 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8076.074132] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8076.074134] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8076.074136] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8076.074139] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8076.074142] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8076.074145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8076.074148] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8076.074156] Call Trace: [ 8076.074161] [ 8076.074166] ? show_regs.cold.9+0x22/0x2f [ 8076.074176] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8076.074183] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8076.074187] ? watchdog+0x40/0x40 [ 8076.074192] ? hrtimer_interrupt+0x128/0x2c0 [ 8076.074196] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8076.074205] ? apic_timer_interrupt+0xf/0x20 [ 8076.074209] [ 8076.074212] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8076.074218] do_raw_spin_lock+0x177/0x190 [ 8076.074222] _raw_spin_lock+0x21/0x30 [ 8076.074227] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8076.079193] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8076.079273] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8076.079280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8076.079283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8076.079296] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8076.079300] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8076.079305] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8076.079308] RDX: 0000000000003767 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8076.079310] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8076.079313] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8076.079315] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8076.079319] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8076.079322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8076.079324] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8076.079334] Call Trace: [ 8076.079340] [ 8076.079346] ? show_regs.cold.9+0x22/0x2f [ 8076.079357] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8076.079363] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8076.079367] ? watchdog+0x40/0x40 [ 8076.079372] ? hrtimer_interrupt+0x128/0x2c0 [ 8076.079376] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8076.079386] ? apic_timer_interrupt+0xf/0x20 [ 8076.079390] [ 8076.079393] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8076.079398] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8076.079403] do_raw_spin_lock+0x177/0x190 [ 8076.079407] _raw_spin_lock+0x21/0x30 [ 8076.079413] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8076.132080] ? do_raw_spin_unlock+0x75/0x190 [ 8076.132115] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8076.132395] ? do_wait_intr+0xf0/0xf0 [ 8076.132403] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8076.149672] obd_mod_free+0x46/0x140 [ptlrpc] [ 8076.150261] mdc_commit_open+0x91/0xb0 [mdc] [ 8076.150300] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8076.152131] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8076.152365] after_reply+0xe2c/0x1780 [ptlrpc] [ 8076.152571] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8076.168281] mdc_close+0x53b/0xd70 [mdc] [ 8076.168339] lmv_close+0x18d/0x490 [lmv] [ 8076.168380] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8076.168859] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8076.168940] ll_md_close+0x274/0x800 [lustre] [ 8076.169022] ll_file_release+0x128/0xc50 [lustre] [ 8076.169104] __fput+0x10d/0x440 [ 8076.169114] ____fput+0x12/0x20 [ 8076.169118] task_work_run+0xc6/0x110 [ 8076.169126] exit_to_usermode_loop+0x1dd/0x1f0 [ 8076.169134] do_syscall_64+0x3d6/0x440 [ 8076.169139] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8076.169148] RIP: 0033:0x7f1a63983cd8 [ 8076.169155] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8076.169158] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8076.169164] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8076.169166] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8076.169168] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8076.169170] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8076.169173] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8076.632965] ? __next_timer_interrupt+0x160/0x160 [ 8076.635785] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8076.640568] ? do_wait_intr+0xf0/0xf0 [ 8076.645963] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8076.652112] kthread+0x1d1/0x200 [ 8076.657336] ? set_kthread_struct+0x70/0x70 [ 8076.659608] ret_from_fork+0x1f/0x30 [ 8096.054488] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 8096.057991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8096.058585] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 8096.101494] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8096.101507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8096.101509] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8096.101521] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8096.101525] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8096.101530] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8096.101532] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8096.101534] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8096.101537] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8096.101539] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8096.101542] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8096.101546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8096.101548] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8096.101556] Call Trace: [ 8096.101561] [ 8096.101567] ? show_regs.cold.9+0x22/0x2f [ 8096.101579] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8096.101586] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8096.101591] ? watchdog+0x40/0x40 [ 8096.101595] ? hrtimer_interrupt+0x128/0x2c0 [ 8096.101599] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8096.101606] ? apic_timer_interrupt+0xf/0x20 [ 8096.101611] [ 8096.101614] ? native_safe_halt+0x2e/0x50 [ 8096.101917] kvm_wait+0xcd/0xe0 [ 8096.101927] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8096.101936] do_raw_spin_lock+0x177/0x190 [ 8096.101940] _raw_spin_lock+0x21/0x30 [ 8096.101944] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8096.102221] ? do_raw_spin_unlock+0x75/0x190 [ 8096.102227] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8096.102634] ? do_wait_intr+0xf0/0xf0 [ 8096.102643] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8096.102853] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8096.103158] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8096.103508] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8096.103631] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8096.103671] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8096.103703] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8096.103733] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8096.103825] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8096.104127] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8096.104160] lmv_intent_open+0x338/0x1150 [lmv] [ 8096.104192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8096.104281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8096.104500] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8096.104526] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8096.104619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8096.104711] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8096.104803] ? do_raw_spin_unlock+0x75/0x190 [ 8096.104810] ? _raw_spin_unlock+0x12/0x30 [ 8096.104817] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8096.104846] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8096.104868] ll_atomic_open+0x36a/0xd70 [lustre] [ 8096.105034] lookup_open+0x356/0xb40 [ 8096.105045] do_last+0x313/0xfc0 [ 8096.105050] ? nd_jump_root+0xe5/0x160 [ 8096.105056] ? path_init+0x437/0x520 [ 8096.105061] path_openat+0xf7/0x500 [ 8096.105066] ? do_raw_spin_unlock+0x75/0x190 [ 8096.105071] do_filp_open+0x99/0x140 [ 8096.105079] ? getname_flags+0x6e/0x330 [ 8096.105085] ? __check_object_size+0xff/0x256 [ 8096.105093] ? do_raw_spin_unlock+0x75/0x190 [ 8096.105097] ? _raw_spin_unlock+0x12/0x30 [ 8096.105103] do_sys_openat2+0x2b4/0x410 [ 8096.105107] do_sys_open+0x73/0xa0 [ 8096.105112] __x64_sys_openat+0x24/0x30 [ 8096.105116] do_syscall_64+0xc1/0x440 [ 8096.105124] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8096.105130] RIP: 0033:0x7f1a63983332 [ 8096.105138] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8096.105143] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8096.105148] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8096.105151] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8096.105153] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8096.105156] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8096.105158] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8096.498335] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8096.539682] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8096.547148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8096.553844] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8096.556793] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8096.571458] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8096.575260] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8096.578899] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8096.581947] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8096.585007] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8096.587931] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8096.590917] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8096.594545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8096.596967] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8096.600035] Call Trace: [ 8096.603833] [ 8096.604577] ? show_regs.cold.9+0x22/0x2f [ 8096.606077] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8096.612864] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8096.614994] ? watchdog+0x40/0x40 [ 8096.617199] ? hrtimer_interrupt+0x128/0x2c0 [ 8096.621838] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8096.626002] ? apic_timer_interrupt+0xf/0x20 [ 8096.630456] [ 8096.631618] ? native_safe_halt+0x2e/0x50 [ 8096.637878] kvm_wait+0xcd/0xe0 [ 8096.640274] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8096.647474] do_raw_spin_lock+0x177/0x190 [ 8096.653751] _raw_spin_lock+0x21/0x30 [ 8096.660716] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8096.669526] ? do_raw_spin_unlock+0x75/0x190 [ 8096.676449] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8096.683826] ? do_wait_intr+0xf0/0xf0 [ 8096.687766] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8096.691075] mdc_close+0x53b/0xd70 [mdc] [ 8096.694065] lmv_close+0x18d/0x490 [lmv] [ 8096.697846] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8096.701550] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8096.706655] ll_md_close+0x274/0x800 [lustre] [ 8096.708868] ll_file_release+0x128/0xc50 [lustre] [ 8096.712827] __fput+0x10d/0x440 [ 8096.713805] ____fput+0x12/0x20 [ 8096.719786] task_work_run+0xc6/0x110 [ 8096.728840] exit_to_usermode_loop+0x1dd/0x1f0 [ 8096.731817] do_syscall_64+0x3d6/0x440 [ 8096.733816] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8096.741539] RIP: 0033:0x7f1a63983cd8 [ 8096.742774] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8096.755756] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8096.763795] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8096.769244] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8096.771172] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8096.774806] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8096.779365] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8100.320919] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6077s! [ 8100.329613] Showing busy workqueues and worker pools: [ 8100.331809] workqueue events: flags=0x0 [ 8100.337716] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8100.345341] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8100.349678] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8100.353685] pending: sec_gc_main [ptlrpc] [ 8100.356287] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8100.359633] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8100.364022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8100.367528] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8100.372159] workqueue events_long: flags=0x0 [ 8100.374151] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8100.377085] pending: lu_objects_flush [obdclass] [ 8100.379683] workqueue events_unbound: flags=0x2 [ 8100.382168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8100.385267] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8100.390433] workqueue events_power_efficient: flags=0x80 [ 8100.393509] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8100.396609] pending: neigh_periodic_work, do_cache_clean [ 8100.399013] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8100.402956] pending: neigh_periodic_work, check_lifetime [ 8100.405615] workqueue mm_percpu_wq: flags=0x8 [ 8100.407629] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8100.411033] pending: vmstat_update [ 8100.413161] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8100.417105] pending: vmstat_update [ 8100.419524] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8100.423130] pending: vmstat_update [ 8100.425109] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8100.428547] pending: vmstat_update [ 8100.430283] workqueue writeback: flags=0x4a [ 8100.432308] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8100.435126] pending: wb_workfn, wb_workfn [ 8100.437166] workqueue xprtiod: flags=0xa [ 8100.440119] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8100.444902] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8100.449561] workqueue ipv6_addrconf: flags=0x40008 [ 8100.452938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8100.456152] pending: addrconf_verify_work [ 8100.458266] workqueue knbd0-recv: flags=0x1a [ 8100.460851] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8100.464486] in-flight: 647:recv_work [ 8100.465848] workqueue ptlrpc_pinger: flags=0x2 [ 8100.467489] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8100.470237] pending: ptlrpc_pinger_main [ptlrpc] [ 8100.472949] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8090s workers=2 idle: 652 [ 8104.056196] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8104.059193] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8104.066180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [ 8104.077633] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8104.077710] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8104.077718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8104.077720] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8104.077734] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8104.077738] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8104.077742] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8104.077745] RDX: 0000000000005174 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8104.077747] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8104.077749] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8104.077751] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8104.077754] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8104.077757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8104.077759] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8104.077768] Call Trace: [ 8104.077773] [ 8104.077779] ? show_regs.cold.9+0x22/0x2f [ 8104.077792] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8104.077798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8104.077802] ? watchdog+0x40/0x40 [ 8104.077806] ? hrtimer_interrupt+0x128/0x2c0 [ 8104.077810] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8104.077818] ? apic_timer_interrupt+0xf/0x20 [ 8104.077822] [ 8104.077825] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8104.077831] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8104.077835] do_raw_spin_lock+0x177/0x190 [ 8104.077839] _raw_spin_lock+0x21/0x30 [ 8104.077845] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8104.078138] obd_mod_free+0x46/0x140 [ptlrpc] [ 8104.078522] mdc_commit_open+0x91/0xb0 [mdc] [ 8104.078562] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8104.078753] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8104.078948] after_reply+0xe2c/0x1780 [ptlrpc] [ 8104.079151] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8104.079477] ? __next_timer_interrupt+0x160/0x160 [ 8104.079487] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8104.079767] ? do_wait_intr+0xf0/0xf0 [ 8104.079777] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8104.079996] kthread+0x1d1/0x200 [ 8104.080003] ? set_kthread_struct+0x70/0x70 [ 8104.080007] ret_from_fork+0x1f/0x30 [ 8104.457459] intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8104.478244] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8104.495633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8104.508716] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8104.516046] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8104.536297] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8104.546837] RAX: 0000000000000e1c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8104.553842] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8104.563083] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8104.572825] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8104.577016] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8104.586376] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8104.599907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8104.610353] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8104.617012] Call Trace: [ 8104.624600] [ 8104.625379] ? show_regs.cold.9+0x22/0x2f [ 8104.628103] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8104.630005] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8104.634602] ? watchdog+0x40/0x40 [ 8104.636645] ? hrtimer_interrupt+0x128/0x2c0 [ 8104.639622] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8104.641346] ? apic_timer_interrupt+0xf/0x20 [ 8104.651136] [ 8104.652227] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8104.658559] do_raw_spin_lock+0x177/0x190 [ 8104.660743] _raw_spin_lock+0x21/0x30 [ 8104.665890] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8104.671153] ? do_raw_spin_unlock+0x75/0x190 [ 8104.673587] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8104.683229] ? do_wait_intr+0xf0/0xf0 [ 8104.684838] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8104.688934] mdc_close+0x53b/0xd70 [mdc] [ 8104.692301] lmv_close+0x18d/0x490 [lmv] [ 8104.696658] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8104.701751] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8104.705238] ll_md_close+0x274/0x800 [lustre] [ 8104.707461] ll_file_release+0x128/0xc50 [lustre] [ 8104.709607] __fput+0x10d/0x440 [ 8104.710991] ____fput+0x12/0x20 [ 8104.712644] task_work_run+0xc6/0x110 [ 8104.714294] exit_to_usermode_loop+0x1dd/0x1f0 [ 8104.716662] do_syscall_64+0x3d6/0x440 [ 8104.718427] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8104.720604] RIP: 0033:0x7f1a63983cd8 [ 8104.722301] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8104.730415] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8104.733702] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8104.736792] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8104.739887] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8104.746820] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8104.751076] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8124.057049] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 8124.060563] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8124.099468] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8124.116851] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8124.131334] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8124.136734] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8124.161536] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8124.172431] RAX: 0000000000004e08 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8124.182300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8124.195855] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8124.202890] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8124.207856] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8124.213094] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8124.225704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8124.232111] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8124.238938] Call Trace: [ 8124.243024] [ 8124.247239] ? show_regs.cold.9+0x22/0x2f [ 8124.256938] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8124.264834] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8124.268804] ? watchdog+0x40/0x40 [ 8124.274271] ? hrtimer_interrupt+0x128/0x2c0 [ 8124.285097] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8124.291801] ? apic_timer_interrupt+0xf/0x20 [ 8124.294739] [ 8124.300766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8124.305539] do_raw_spin_lock+0x177/0x190 [ 8124.309080] _raw_spin_lock+0x21/0x30 [ 8124.310964] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8124.316147] ? do_raw_spin_unlock+0x75/0x190 [ 8124.318145] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8124.326836] ? do_wait_intr+0xf0/0xf0 [ 8124.329088] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8124.331980] mdc_close+0x53b/0xd70 [mdc] [ 8124.336952] lmv_close+0x18d/0x490 [lmv] [ 8124.341846] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8124.345040] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8124.351401] ll_md_close+0x274/0x800 [lustre] [ 8124.356470] ll_file_release+0x128/0xc50 [lustre] [ 8124.370229] __fput+0x10d/0x440 [ 8124.371364] ____fput+0x12/0x20 [ 8124.372437] task_work_run+0xc6/0x110 [ 8124.381813] exit_to_usermode_loop+0x1dd/0x1f0 [ 8124.383356] do_syscall_64+0x3d6/0x440 [ 8124.389516] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8124.400483] RIP: 0033:0x7f1a63983cd8 [ 8124.409069] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8124.425278] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8124.430492] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8124.440026] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8124.445686] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8124.449100] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8124.452845] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8128.053027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8128.058512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8128.121818] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8128.132440] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8128.145191] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8128.147097] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8128.163645] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8128.186909] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8128.200413] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8128.208392] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8128.215519] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8128.223812] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8128.237547] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8128.252331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8128.254430] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8128.264601] Call Trace: [ 8128.265489] [ 8128.270212] ? show_regs.cold.9+0x22/0x2f [ 8128.271762] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8128.276955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8128.283590] ? watchdog+0x40/0x40 [ 8128.286037] ? hrtimer_interrupt+0x128/0x2c0 [ 8128.287647] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8128.297773] ? apic_timer_interrupt+0xf/0x20 [ 8128.302595] [ 8128.304985] ? native_safe_halt+0x2e/0x50 [ 8128.309874] kvm_wait+0xcd/0xe0 [ 8128.312032] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8128.329513] do_raw_spin_lock+0x177/0x190 [ 8128.331217] _raw_spin_lock+0x21/0x30 [ 8128.336142] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8128.344701] ? do_raw_spin_unlock+0x75/0x190 [ 8128.354179] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8128.360986] ? do_wait_intr+0xf0/0xf0 [ 8128.370571] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8128.378931] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8128.389878] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8128.396611] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8128.401774] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8128.414550] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8128.418342] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8128.424767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8128.430479] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8128.438267] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8128.444708] lmv_intent_open+0x338/0x1150 [lmv] [ 8128.447899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8128.454240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8128.465890] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8128.471903] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8128.483600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8128.492830] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8128.495748] ? do_raw_spin_unlock+0x75/0x190 [ 8128.500711] ? _raw_spin_unlock+0x12/0x30 [ 8128.503697] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8128.506466] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8128.511320] ll_atomic_open+0x36a/0xd70 [lustre] [ 8128.516523] lookup_open+0x356/0xb40 [ 8128.518919] do_last+0x313/0xfc0 [ 8128.520068] ? nd_jump_root+0xe5/0x160 [ 8128.523821] ? path_init+0x437/0x520 [ 8128.527258] path_openat+0xf7/0x500 [ 8128.528855] ? do_raw_spin_unlock+0x75/0x190 [ 8128.532316] do_filp_open+0x99/0x140 [ 8128.539040] ? getname_flags+0x6e/0x330 [ 8128.542982] ? __check_object_size+0xff/0x256 [ 8128.546580] ? do_raw_spin_unlock+0x75/0x190 [ 8128.550650] ? _raw_spin_unlock+0x12/0x30 [ 8128.555672] do_sys_openat2+0x2b4/0x410 [ 8128.557997] do_sys_open+0x73/0xa0 [ 8128.559831] __x64_sys_openat+0x24/0x30 [ 8128.564519] do_syscall_64+0xc1/0x440 [ 8128.568194] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8128.574954] RIP: 0033:0x7f1a63983332 [ 8128.576045] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8128.594785] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8128.601741] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8128.605477] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8128.608871] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8128.613630] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8128.617245] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8130.532059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6107s! [ 8130.535713] Showing busy workqueues and worker pools: [ 8130.538474] workqueue events: flags=0x0 [ 8130.540553] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8130.543554] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8130.547419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8130.550652] pending: sec_gc_main [ptlrpc] [ 8130.552814] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8130.560381] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8130.568472] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8130.572413] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8130.579602] workqueue events_long: flags=0x0 [ 8130.581845] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8130.589548] pending: lu_objects_flush [obdclass] [ 8130.592991] workqueue events_unbound: flags=0x2 [ 8130.598891] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8130.605891] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8130.617296] workqueue events_power_efficient: flags=0x80 [ 8130.624604] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8130.631626] pending: neigh_periodic_work, do_cache_clean [ 8130.640475] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8130.648678] pending: neigh_periodic_work, check_lifetime [ 8130.653556] workqueue mm_percpu_wq: flags=0x8 [ 8130.655844] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8130.662582] pending: vmstat_update [ 8130.667576] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8130.673393] pending: vmstat_update [ 8130.676403] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8130.683221] pending: vmstat_update [ 8130.689584] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8130.694734] pending: vmstat_update [ 8130.698278] workqueue writeback: flags=0x4a [ 8130.700904] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8130.706469] pending: wb_workfn, wb_workfn [ 8130.715460] workqueue xprtiod: flags=0xa [ 8130.721026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8130.724646] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8130.728821] workqueue ipv6_addrconf: flags=0x40008 [ 8130.733519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8130.741221] pending: addrconf_verify_work [ 8130.743942] workqueue knbd0-recv: flags=0x1a [ 8130.747329] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8130.753414] in-flight: 647:recv_work [ 8130.754784] workqueue ptlrpc_pinger: flags=0x2 [ 8130.757818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8130.762343] pending: ptlrpc_pinger_main [ptlrpc] [ 8130.764432] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8120s workers=2 idle: 652 [ 8132.056260] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8132.058287] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8132.060575] Modules linked in: lustre(O) [ 8132.064275] Modules linked in: lustre(O) osp(O) ofd(O) [ 8132.066038] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 8132.070713] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8132.070794] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8132.070801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8132.070804] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8132.070818] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8132.070822] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8132.070826] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8132.070829] RDX: 0000000000003d58 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8132.070831] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8132.070834] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8132.070836] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8132.070839] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8132.070846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8132.070848] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8132.070856] Call Trace: [ 8132.070862] [ 8132.070873] ? show_regs.cold.9+0x22/0x2f [ 8132.070887] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8132.070893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8132.070897] ? watchdog+0x40/0x40 [ 8132.070901] ? hrtimer_interrupt+0x128/0x2c0 [ 8132.070906] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8132.070915] ? apic_timer_interrupt+0xf/0x20 [ 8132.070919] [ 8132.070922] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8132.070926] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8132.070931] do_raw_spin_lock+0x177/0x190 [ 8132.070935] _raw_spin_lock+0x21/0x30 [ 8132.071271] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8132.099142] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 [ 8132.125969] obd_mod_free+0x46/0x140 [ptlrpc] [ 8132.137594] squashfs [ 8132.143291] mdc_commit_open+0x91/0xb0 [mdc] [ 8132.152342] crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8132.152373] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8132.163868] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8132.164520] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8132.164938] after_reply+0xe2c/0x1780 [ptlrpc] [ 8132.165167] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8132.165446] ? __next_timer_interrupt+0x160/0x160 [ 8132.165456] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8132.165750] ? do_wait_intr+0xf0/0xf0 [ 8132.165760] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8132.165987] kthread+0x1d1/0x200 [ 8132.165992] ? set_kthread_struct+0x70/0x70 [ 8132.165996] ret_from_fork+0x1f/0x30 [ 8132.317819] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8132.321933] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8132.327793] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8132.343842] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8132.351741] RAX: 0000000000007204 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8132.357326] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8132.361237] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8132.364525] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8132.367409] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8132.370584] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8132.374537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8132.376816] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8132.379763] Call Trace: [ 8132.380807] [ 8132.382877] ? show_regs.cold.9+0x22/0x2f [ 8132.385597] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8132.387374] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8132.390464] ? watchdog+0x40/0x40 [ 8132.391682] ? hrtimer_interrupt+0x128/0x2c0 [ 8132.393633] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8132.395700] ? apic_timer_interrupt+0xf/0x20 [ 8132.397658] [ 8132.398701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8132.404999] do_raw_spin_lock+0x177/0x190 [ 8132.407241] _raw_spin_lock+0x21/0x30 [ 8132.409250] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8132.412706] ? do_raw_spin_unlock+0x75/0x190 [ 8132.416140] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8132.418598] ? do_wait_intr+0xf0/0xf0 [ 8132.420406] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8132.423430] mdc_close+0x53b/0xd70 [mdc] [ 8132.425607] lmv_close+0x18d/0x490 [lmv] [ 8132.427452] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8132.429788] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8132.431865] ll_md_close+0x274/0x800 [lustre] [ 8132.434240] ll_file_release+0x128/0xc50 [lustre] [ 8132.436712] __fput+0x10d/0x440 [ 8132.438990] ____fput+0x12/0x20 [ 8132.441647] task_work_run+0xc6/0x110 [ 8132.442873] exit_to_usermode_loop+0x1dd/0x1f0 [ 8132.446546] do_syscall_64+0x3d6/0x440 [ 8132.449757] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8132.452207] RIP: 0033:0x7f1a63983cd8 [ 8132.454739] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8132.467523] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8132.470629] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8132.475108] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8132.478411] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8132.482759] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8132.487351] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8152.058760] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 8152.063756] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8152.096396] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8152.105409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8152.109740] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8152.114022] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8152.123424] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8152.129401] RAX: 0000000000007d14 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8152.134323] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8152.140584] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8152.147465] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8152.156224] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8152.160280] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8152.168644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8152.174720] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8152.177581] Call Trace: [ 8152.178825] [ 8152.179599] ? show_regs.cold.9+0x22/0x2f [ 8152.181569] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8152.183672] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8152.185431] ? watchdog+0x40/0x40 [ 8152.186410] ? hrtimer_interrupt+0x128/0x2c0 [ 8152.194273] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8152.197390] ? apic_timer_interrupt+0xf/0x20 [ 8152.200499] [ 8152.201297] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8152.203722] do_raw_spin_lock+0x177/0x190 [ 8152.210587] _raw_spin_lock+0x21/0x30 [ 8152.213528] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8152.215603] ? do_raw_spin_unlock+0x75/0x190 [ 8152.222681] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8152.225603] ? do_wait_intr+0xf0/0xf0 [ 8152.228478] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8152.239529] mdc_close+0x53b/0xd70 [mdc] [ 8152.241138] lmv_close+0x18d/0x490 [lmv] [ 8152.247477] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8152.254359] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8152.256537] ll_md_close+0x274/0x800 [lustre] [ 8152.259837] ll_file_release+0x128/0xc50 [lustre] [ 8152.267277] __fput+0x10d/0x440 [ 8152.272637] ____fput+0x12/0x20 [ 8152.273758] task_work_run+0xc6/0x110 [ 8152.287428] exit_to_usermode_loop+0x1dd/0x1f0 [ 8152.288951] do_syscall_64+0x3d6/0x440 [ 8152.292592] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8152.296635] RIP: 0033:0x7f1a63983cd8 [ 8152.300003] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8152.319049] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8152.327598] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8152.332224] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8152.340760] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8152.347450] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8152.355884] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8156.055118] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8156.063226] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8156.112178] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8156.124459] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8156.132904] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8156.135960] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8156.154006] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8156.160092] RAX: 0000000000007f86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8156.169683] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8156.173656] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8156.181878] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8156.188145] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8156.193458] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8156.200472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8156.204510] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8156.208682] Call Trace: [ 8156.209938] [ 8156.211008] ? show_regs.cold.9+0x22/0x2f [ 8156.218449] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8156.224513] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8156.227674] ? watchdog+0x40/0x40 [ 8156.233910] ? hrtimer_interrupt+0x128/0x2c0 [ 8156.239911] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8156.242266] ? apic_timer_interrupt+0xf/0x20 [ 8156.250301] [ 8156.251336] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8156.259165] do_raw_spin_lock+0x177/0x190 [ 8156.260604] _raw_spin_lock+0x21/0x30 [ 8156.264023] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8156.270316] ? do_raw_spin_unlock+0x75/0x190 [ 8156.271947] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8156.274874] ? do_wait_intr+0xf0/0xf0 [ 8156.276511] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8156.278743] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8156.281571] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8156.284483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8156.286779] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8156.289108] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8156.291240] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8156.293577] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8156.296204] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8156.298945] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8156.302283] lmv_intent_open+0x338/0x1150 [lmv] [ 8156.305350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8156.308952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8156.312222] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8156.314992] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8156.318270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8156.321319] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8156.324413] ? do_raw_spin_unlock+0x75/0x190 [ 8156.334413] ? _raw_spin_unlock+0x12/0x30 [ 8156.340260] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8156.345370] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8156.352441] ll_atomic_open+0x36a/0xd70 [lustre] [ 8156.358833] lookup_open+0x356/0xb40 [ 8156.364287] do_last+0x313/0xfc0 [ 8156.366220] ? nd_jump_root+0xe5/0x160 [ 8156.368794] ? path_init+0x437/0x520 [ 8156.374056] path_openat+0xf7/0x500 [ 8156.375705] ? do_raw_spin_unlock+0x75/0x190 [ 8156.378118] do_filp_open+0x99/0x140 [ 8156.382894] ? getname_flags+0x6e/0x330 [ 8156.384762] ? __check_object_size+0xff/0x256 [ 8156.391433] ? do_raw_spin_unlock+0x75/0x190 [ 8156.395414] ? _raw_spin_unlock+0x12/0x30 [ 8156.397738] do_sys_openat2+0x2b4/0x410 [ 8156.399161] do_sys_open+0x73/0xa0 [ 8156.401962] __x64_sys_openat+0x24/0x30 [ 8156.406936] do_syscall_64+0xc1/0x440 [ 8156.412196] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8156.414032] RIP: 0033:0x7f1a63983332 [ 8156.416660] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8156.428005] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8156.431045] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8156.434115] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8156.438327] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8156.444765] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8156.455775] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8160.056556] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8160.059006] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8160.061105] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8160.077953] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8160.080533] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8160.080613] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8160.080621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8160.080625] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8160.080638] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8160.080642] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8160.080647] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8160.080649] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8160.080651] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8160.080654] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8160.080657] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8160.080660] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8160.080663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8160.080665] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8160.080673] Call Trace: [ 8160.080677] [ 8160.080682] ? show_regs.cold.9+0x22/0x2f [ 8160.080691] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8160.080696] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8160.080700] ? watchdog+0x40/0x40 [ 8160.080704] ? hrtimer_interrupt+0x128/0x2c0 [ 8160.080708] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8160.080714] ? apic_timer_interrupt+0xf/0x20 [ 8160.080718] [ 8160.080720] ? native_safe_halt+0x2e/0x50 [ 8160.080724] kvm_wait+0xcd/0xe0 [ 8160.080729] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8160.080736] do_raw_spin_lock+0x177/0x190 [ 8160.080739] _raw_spin_lock+0x21/0x30 [ 8160.080744] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8160.081034] obd_mod_free+0x46/0x140 [ptlrpc] [ 8160.081229] mdc_commit_open+0x91/0xb0 [mdc] [ 8160.081272] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8160.081464] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8160.081657] after_reply+0xe2c/0x1780 [ptlrpc] [ 8160.081862] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8160.082054] ? __next_timer_interrupt+0x160/0x160 [ 8160.082063] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8160.082264] ? do_wait_intr+0xf0/0xf0 [ 8160.082272] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8160.087525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8160.087535] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8160.087551] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8160.087556] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8160.087561] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8160.087563] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8160.087566] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8160.087568] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8160.087571] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8160.087574] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8160.087578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8160.087580] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8160.087589] Call Trace: [ 8160.087595] [ 8160.087600] ? show_regs.cold.9+0x22/0x2f [ 8160.087611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8160.087617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8160.087622] ? watchdog+0x40/0x40 [ 8160.087626] ? hrtimer_interrupt+0x128/0x2c0 [ 8160.087631] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8160.087638] ? apic_timer_interrupt+0xf/0x20 [ 8160.087643] [ 8160.087646] ? native_safe_halt+0x2e/0x50 [ 8160.087650] kvm_wait+0xcd/0xe0 [ 8160.087658] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8160.087666] do_raw_spin_lock+0x177/0x190 [ 8160.087671] _raw_spin_lock+0x21/0x30 [ 8160.087674] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8160.088142] ? do_raw_spin_unlock+0x75/0x190 [ 8160.088148] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8160.088342] ? do_wait_intr+0xf0/0xf0 [ 8160.088351] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8160.088619] mdc_close+0x53b/0xd70 [mdc] [ 8160.088665] lmv_close+0x18d/0x490 [lmv] [ 8160.088703] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8160.129337] kthread+0x1d1/0x200 [ 8160.129356] ? set_kthread_struct+0x70/0x70 [ 8160.129361] ret_from_fork+0x1f/0x30 [ 8160.489993] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8160.493554] ll_md_close+0x274/0x800 [lustre] [ 8160.499414] ll_file_release+0x128/0xc50 [lustre] [ 8160.507424] __fput+0x10d/0x440 [ 8160.511946] ____fput+0x12/0x20 [ 8160.515191] task_work_run+0xc6/0x110 [ 8160.517307] exit_to_usermode_loop+0x1dd/0x1f0 [ 8160.521004] do_syscall_64+0x3d6/0x440 [ 8160.524737] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8160.529144] RIP: 0033:0x7f1a63983cd8 [ 8160.532033] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8160.542576] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8160.548376] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8160.559938] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8160.565433] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8160.574135] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8160.579175] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8161.249395] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6138s! [ 8161.263587] Showing busy workqueues and worker pools: [ 8161.266631] workqueue events: flags=0x0 [ 8161.267956] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8161.275555] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8161.281876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8161.285618] pending: sec_gc_main [ptlrpc] [ 8161.288223] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8161.291753] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8161.297434] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8161.301552] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8161.309521] workqueue events_long: flags=0x0 [ 8161.313512] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8161.315698] pending: lu_objects_flush [obdclass] [ 8161.322126] workqueue events_unbound: flags=0x2 [ 8161.323494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8161.337920] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8161.345629] workqueue events_power_efficient: flags=0x80 [ 8161.349438] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8161.351922] pending: neigh_periodic_work, do_cache_clean [ 8161.355556] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8161.359310] pending: neigh_periodic_work, check_lifetime [ 8161.362612] workqueue mm_percpu_wq: flags=0x8 [ 8161.370334] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8161.378557] pending: vmstat_update [ 8161.381047] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8161.382863] pending: vmstat_update [ 8161.391453] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8161.396857] pending: vmstat_update [ 8161.398654] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8161.400632] pending: vmstat_update [ 8161.402403] workqueue writeback: flags=0x4a [ 8161.404349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8161.407383] pending: wb_workfn, wb_workfn [ 8161.409277] workqueue xprtiod: flags=0xa [ 8161.410612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8161.417910] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8161.425274] workqueue ipv6_addrconf: flags=0x40008 [ 8161.427547] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8161.434559] pending: addrconf_verify_work [ 8161.442292] workqueue knbd0-recv: flags=0x1a [ 8161.447116] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8161.458402] in-flight: 647:recv_work [ 8161.460409] workqueue ptlrpc_pinger: flags=0x2 [ 8161.462027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8161.465536] pending: ptlrpc_pinger_main [ptlrpc] [ 8161.471663] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8151s workers=2 idle: 652 [ 8180.057091] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 8180.060718] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8180.099350] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8180.104256] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8180.108343] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8180.110386] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8180.125951] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8180.130987] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8180.135239] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8180.139249] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8180.146326] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8180.150540] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8180.160530] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8180.167789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8180.173266] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8180.177373] Call Trace: [ 8180.178568] [ 8180.179803] ? show_regs.cold.9+0x22/0x2f [ 8180.182091] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8180.186131] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8180.188832] ? watchdog+0x40/0x40 [ 8180.191713] ? hrtimer_interrupt+0x128/0x2c0 [ 8180.200145] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8180.202861] ? apic_timer_interrupt+0xf/0x20 [ 8180.207404] [ 8180.213275] ? native_safe_halt+0x2e/0x50 [ 8180.220108] kvm_wait+0xcd/0xe0 [ 8180.226072] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8180.237133] do_raw_spin_lock+0x177/0x190 [ 8180.241348] _raw_spin_lock+0x21/0x30 [ 8180.252242] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8180.257156] ? do_raw_spin_unlock+0x75/0x190 [ 8180.265127] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8180.269348] ? do_wait_intr+0xf0/0xf0 [ 8180.271395] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8180.278239] mdc_close+0x53b/0xd70 [mdc] [ 8180.282088] lmv_close+0x18d/0x490 [lmv] [ 8180.284653] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8180.287720] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8180.291801] ll_md_close+0x274/0x800 [lustre] [ 8180.294568] ll_file_release+0x128/0xc50 [lustre] [ 8180.299022] __fput+0x10d/0x440 [ 8180.313460] ____fput+0x12/0x20 [ 8180.314729] task_work_run+0xc6/0x110 [ 8180.315675] exit_to_usermode_loop+0x1dd/0x1f0 [ 8180.322188] do_syscall_64+0x3d6/0x440 [ 8180.325035] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8180.329494] RIP: 0033:0x7f1a63983cd8 [ 8180.331289] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8180.340747] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8180.343923] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8180.346950] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8180.349961] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8180.353193] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8180.356329] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8188.053140] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8188.056872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8188.059147] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8188.064104] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8188.064119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8188.064196] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8188.064203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8188.064206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8188.064220] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8188.064225] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8188.064230] RAX: 000000000000714b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8188.064233] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8188.064235] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8188.064238] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8188.064241] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8188.064244] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8188.064248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8188.064250] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8188.064258] Call Trace: [ 8188.064263] [ 8188.064269] ? show_regs.cold.9+0x22/0x2f [ 8188.064280] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8188.064288] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8188.064292] ? watchdog+0x40/0x40 [ 8188.064297] ? hrtimer_interrupt+0x128/0x2c0 [ 8188.064301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8188.064309] ? apic_timer_interrupt+0xf/0x20 [ 8188.064313] [ 8188.064316] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8188.064322] do_raw_spin_lock+0x177/0x190 [ 8188.064326] _raw_spin_lock+0x21/0x30 [ 8188.064330] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8188.066591] ? do_raw_spin_unlock+0x75/0x190 [ 8188.066602] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8188.067138] ? do_wait_intr+0xf0/0xf0 [ 8188.067147] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8188.067345] mdc_close+0x53b/0xd70 [mdc] [ 8188.068305] lmv_close+0x18d/0x490 [lmv] [ 8188.068335] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8188.068748] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8188.069309] ll_md_close+0x274/0x800 [lustre] [ 8188.069851] ll_file_release+0x128/0xc50 [lustre] [ 8188.069938] __fput+0x10d/0x440 [ 8188.069949] ____fput+0x12/0x20 [ 8188.069954] task_work_run+0xc6/0x110 [ 8188.069963] exit_to_usermode_loop+0x1dd/0x1f0 [ 8188.069970] do_syscall_64+0x3d6/0x440 [ 8188.069976] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8188.069985] RIP: 0033:0x7f1a63983cd8 [ 8188.069993] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8188.069998] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8188.070004] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8188.070006] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8188.070009] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8188.070011] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8188.070014] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8188.080226] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8188.080244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8188.080247] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8188.080266] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8188.080270] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8188.080276] RAX: 000000000000721f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8188.080279] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8188.080281] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8188.080283] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8188.080285] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8188.080289] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8188.080292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8188.080294] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8188.080314] Call Trace: [ 8188.080323] [ 8188.080331] ? show_regs.cold.9+0x22/0x2f [ 8188.080341] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8188.080346] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8188.080351] ? watchdog+0x40/0x40 [ 8188.080355] ? hrtimer_interrupt+0x128/0x2c0 [ 8188.080359] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8188.080367] ? apic_timer_interrupt+0xf/0x20 [ 8188.080372] [ 8188.080375] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8188.080382] do_raw_spin_lock+0x177/0x190 [ 8188.080387] _raw_spin_lock+0x21/0x30 [ 8188.080391] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8188.080679] ? do_raw_spin_unlock+0x75/0x190 [ 8188.080685] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8188.088850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8188.088931] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8188.088939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8188.088942] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8188.088957] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8188.088962] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8188.088967] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8188.088969] RDX: 00000000000008f7 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8188.088972] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8188.088974] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8188.088977] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8188.088980] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8188.088983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8188.088985] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8188.088994] Call Trace: [ 8188.088999] [ 8188.089004] ? show_regs.cold.9+0x22/0x2f [ 8188.089013] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8188.089019] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8188.089024] ? watchdog+0x40/0x40 [ 8188.089028] ? hrtimer_interrupt+0x128/0x2c0 [ 8188.089032] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8188.089040] ? apic_timer_interrupt+0xf/0x20 [ 8188.089044] [ 8188.089047] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8188.089051] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8188.089055] do_raw_spin_lock+0x177/0x190 [ 8188.089059] _raw_spin_lock+0x21/0x30 [ 8188.089065] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8188.089405] obd_mod_free+0x46/0x140 [ptlrpc] [ 8188.089625] mdc_commit_open+0x91/0xb0 [mdc] [ 8188.089672] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8188.089869] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8188.094253] ? do_wait_intr+0xf0/0xf0 [ 8188.094279] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8188.094714] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8188.095011] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8188.095314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8188.095443] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8188.095529] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8188.095562] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8188.097213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8188.120724] after_reply+0xe2c/0x1780 [ptlrpc] [ 8188.127066] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8188.127491] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8188.127615] lmv_intent_open+0x338/0x1150 [lmv] [ 8188.127782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8188.128237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8188.128345] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8188.128370] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8188.128471] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8188.128586] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8188.128692] ? do_raw_spin_unlock+0x75/0x190 [ 8188.128700] ? _raw_spin_unlock+0x12/0x30 [ 8188.128706] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8188.128733] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8188.128753] ll_atomic_open+0x36a/0xd70 [lustre] [ 8188.132855] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8188.133168] ? __next_timer_interrupt+0x160/0x160 [ 8188.133179] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8188.133579] ? do_wait_intr+0xf0/0xf0 [ 8188.133589] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8188.134060] kthread+0x1d1/0x200 [ 8188.134068] ? set_kthread_struct+0x70/0x70 [ 8188.134071] ret_from_fork+0x1f/0x30 [ 8188.569817] lookup_open+0x356/0xb40 [ 8188.572072] do_last+0x313/0xfc0 [ 8188.573399] ? nd_jump_root+0xe5/0x160 [ 8188.575880] ? path_init+0x437/0x520 [ 8188.578418] path_openat+0xf7/0x500 [ 8188.579941] ? do_raw_spin_unlock+0x75/0x190 [ 8188.581928] do_filp_open+0x99/0x140 [ 8188.583732] ? getname_flags+0x6e/0x330 [ 8188.585615] ? __check_object_size+0xff/0x256 [ 8188.587954] ? do_raw_spin_unlock+0x75/0x190 [ 8188.590432] ? _raw_spin_unlock+0x12/0x30 [ 8188.592459] do_sys_openat2+0x2b4/0x410 [ 8188.594461] do_sys_open+0x73/0xa0 [ 8188.596049] __x64_sys_openat+0x24/0x30 [ 8188.598128] do_syscall_64+0xc1/0x440 [ 8188.600315] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8188.602760] RIP: 0033:0x7f1a63983332 [ 8188.604654] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8188.613755] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8188.617567] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8188.621079] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8188.624283] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8188.627931] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8188.632023] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8191.970072] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6168s! [ 8191.973128] Showing busy workqueues and worker pools: [ 8191.981068] workqueue events: flags=0x0 [ 8191.985191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8191.995101] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8192.003755] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8192.010326] pending: sec_gc_main [ptlrpc] [ 8192.016856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8192.023913] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8192.036525] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8192.041686] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8192.053095] workqueue events_long: flags=0x0 [ 8192.055571] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8192.058811] pending: lu_objects_flush [obdclass] [ 8192.064372] workqueue events_unbound: flags=0x2 [ 8192.066634] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8192.078407] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8192.084716] workqueue events_power_efficient: flags=0x80 [ 8192.087347] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8192.091657] pending: neigh_periodic_work, do_cache_clean [ 8192.095012] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8192.099272] pending: neigh_periodic_work, check_lifetime [ 8192.102376] workqueue mm_percpu_wq: flags=0x8 [ 8192.104021] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8192.112322] pending: vmstat_update [ 8192.121891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8192.125167] pending: vmstat_update [ 8192.129273] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8192.132416] pending: vmstat_update [ 8192.135840] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8192.146200] pending: vmstat_update [ 8192.152135] workqueue writeback: flags=0x4a [ 8192.153609] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8192.161559] pending: wb_workfn, wb_workfn [ 8192.163248] workqueue xprtiod: flags=0xa [ 8192.166440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8192.169815] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8192.179837] workqueue ipv6_addrconf: flags=0x40008 [ 8192.183112] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8192.186686] pending: addrconf_verify_work [ 8192.188307] workqueue knbd0-recv: flags=0x1a [ 8192.191576] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8192.199035] in-flight: 647:recv_work [ 8192.200546] workqueue ptlrpc_pinger: flags=0x2 [ 8192.208407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8192.214367] pending: ptlrpc_pinger_main [ptlrpc] [ 8192.218129] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8182s workers=2 idle: 652 [ 8202.566023] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 8202.568656] rcu: 3-...!: (3893257 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 8202.585669] (detected by 1, t=6180160 jiffies, g=269633, q=2244) [ 8202.595518] Sending NMI from CPU 1 to CPUs 3: [ 8202.607778] NMI backtrace for cpu 3 [ 8202.607786] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8202.607788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8202.607789] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8202.607791] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8202.607793] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 8202.607796] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8202.607798] RDX: 00000000000008fc RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8202.607799] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8202.607800] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8202.607801] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8202.607802] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8202.607803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8202.607804] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8202.607805] Call Trace: [ 8202.607806] [ 8202.607807] ? show_regs.cold.9+0x22/0x2f [ 8202.607808] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8202.607809] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8202.607810] ? nmi_handle+0xbd/0x210 [ 8202.607811] ? default_do_nmi+0x4b/0x1b0 [ 8202.607812] ? do_nmi+0x280/0x380 [ 8202.607812] ? end_repeat_nmi+0x16/0x1f [ 8202.607814] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8202.607815] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8202.607816] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8202.607817] [ 8202.607817] do_raw_spin_lock+0x177/0x190 [ 8202.607818] _raw_spin_lock+0x21/0x30 [ 8202.607820] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8202.607821] obd_mod_free+0x46/0x140 [ptlrpc] [ 8202.607822] mdc_commit_open+0x91/0xb0 [mdc] [ 8202.607823] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8202.607824] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8202.607825] after_reply+0xe2c/0x1780 [ptlrpc] [ 8202.607826] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8202.607827] ? __next_timer_interrupt+0x160/0x160 [ 8202.607828] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8202.607829] ? do_wait_intr+0xf0/0xf0 [ 8202.607830] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8202.607831] kthread+0x1d1/0x200 [ 8202.607832] ? set_kthread_struct+0x70/0x70 [ 8202.607833] ret_from_fork+0x1f/0x30 [ 8202.611102] rcu: rcu_sched kthread starved for 6179806 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8202.819824] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8202.827189] rcu: RCU grace-period kthread stack dump: [ 8202.832935] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8202.845041] Call Trace: [ 8202.850033] __schedule+0x351/0xcb0 [ 8202.851377] schedule+0xc0/0x180 [ 8202.862210] schedule_timeout+0xb4/0x190 [ 8202.863895] ? do_raw_spin_unlock+0x75/0x190 [ 8202.874674] ? __next_timer_interrupt+0x160/0x160 [ 8202.887511] ? prepare_to_swait_event+0xf3/0x140 [ 8202.902129] rcu_gp_kthread+0x8dd/0x15b0 [ 8202.908779] ? force_qs_rnp+0x2c0/0x2c0 [ 8202.920505] kthread+0x1d1/0x200 [ 8202.922012] ? set_kthread_struct+0x70/0x70 [ 8202.935727] ret_from_fork+0x1f/0x30 [ 8202.937540] rcu: Stack dump where RCU GP kthread last ran: [ 8202.947214] Sending NMI from CPU 1 to CPUs 2: [ 8202.953079] NMI backtrace for cpu 2 [ 8202.953086] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8202.953087] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8202.953089] RIP: 0010:kvm_wait+0x1b/0xe0 [ 8202.953090] Code: 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 65 8b 05 5c f7 d8 70 a9 00 00 f0 00 75 74 48 83 05 0d 53 e6 02 01 <9c> 58 0f 1f 44 00 00 f6 c4 02 75 11 0f b6 07 48 83 05 fe 52 e6 02 [ 8202.953091] RSP: 0018:ffffc3854145fae8 EFLAGS: 00000202 [ 8202.953093] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8202.953094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8202.953095] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8202.953096] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8202.953096] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8202.953097] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8202.953098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8202.953099] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8202.953099] Call Trace: [ 8202.953100] [ 8202.953101] ? show_regs.cold.9+0x22/0x2f [ 8202.953101] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8202.953102] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8202.953103] ? nmi_handle+0xbd/0x210 [ 8202.953104] ? default_do_nmi+0x4b/0x1b0 [ 8202.953104] ? do_nmi+0x280/0x380 [ 8202.953105] ? end_repeat_nmi+0x16/0x1f [ 8202.953105] ? kvm_wait+0x1b/0xe0 [ 8202.953106] ? kvm_wait+0x1b/0xe0 [ 8202.953106] ? kvm_wait+0x1b/0xe0 [ 8202.953107] [ 8202.953107] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8202.953108] do_raw_spin_lock+0x177/0x190 [ 8202.953108] _raw_spin_lock+0x21/0x30 [ 8202.953109] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8202.953109] ? do_raw_spin_unlock+0x75/0x190 [ 8202.953110] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8202.953111] ? do_wait_intr+0xf0/0xf0 [ 8202.953111] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8202.953112] mdc_close+0x53b/0xd70 [mdc] [ 8202.953112] lmv_close+0x18d/0x490 [lmv] [ 8202.953113] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8202.953114] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8202.953115] ll_md_close+0x274/0x800 [lustre] [ 8202.953115] ll_file_release+0x128/0xc50 [lustre] [ 8202.953116] __fput+0x10d/0x440 [ 8202.953117] ____fput+0x12/0x20 [ 8202.953117] task_work_run+0xc6/0x110 [ 8202.953118] exit_to_usermode_loop+0x1dd/0x1f0 [ 8202.953118] do_syscall_64+0x3d6/0x440 [ 8202.953119] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8202.953120] RIP: 0033:0x7f1a63983cd8 [ 8202.953120] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8202.953122] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8202.953123] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8202.953124] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8202.953125] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8202.953125] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8202.953126] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8220.053010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8220.056981] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8220.100369] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8220.110454] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8220.118569] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8220.128950] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8220.143910] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8220.157015] RAX: 00000000000018d3 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8220.168089] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8220.170803] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8220.191160] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8220.199999] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8220.203516] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8220.210120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8220.212274] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8220.224814] Call Trace: [ 8220.228799] [ 8220.229897] ? show_regs.cold.9+0x22/0x2f [ 8220.235447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8220.246130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8220.248451] ? watchdog+0x40/0x40 [ 8220.249859] ? hrtimer_interrupt+0x128/0x2c0 [ 8220.267491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8220.274009] ? apic_timer_interrupt+0xf/0x20 [ 8220.283227] [ 8220.284119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8220.286846] do_raw_spin_lock+0x177/0x190 [ 8220.294659] _raw_spin_lock+0x21/0x30 [ 8220.300560] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8220.309714] ? do_raw_spin_unlock+0x75/0x190 [ 8220.317177] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8220.320357] ? do_wait_intr+0xf0/0xf0 [ 8220.322800] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8220.326714] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8220.329078] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8220.332635] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8220.336009] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8220.345778] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8220.350849] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8220.358971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8220.365893] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8220.369309] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8220.372305] lmv_intent_open+0x338/0x1150 [lmv] [ 8220.374748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8220.377351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8220.380690] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8220.386628] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8220.389123] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8220.393366] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8220.402984] ? do_raw_spin_unlock+0x75/0x190 [ 8220.404338] ? _raw_spin_unlock+0x12/0x30 [ 8220.405397] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8220.415324] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8220.418128] ll_atomic_open+0x36a/0xd70 [lustre] [ 8220.423372] lookup_open+0x356/0xb40 [ 8220.424894] do_last+0x313/0xfc0 [ 8220.426123] ? nd_jump_root+0xe5/0x160 [ 8220.434334] ? path_init+0x437/0x520 [ 8220.437724] path_openat+0xf7/0x500 [ 8220.441749] ? do_raw_spin_unlock+0x75/0x190 [ 8220.446438] do_filp_open+0x99/0x140 [ 8220.448689] ? getname_flags+0x6e/0x330 [ 8220.450065] ? __check_object_size+0xff/0x256 [ 8220.452879] ? do_raw_spin_unlock+0x75/0x190 [ 8220.456622] ? _raw_spin_unlock+0x12/0x30 [ 8220.459872] do_sys_openat2+0x2b4/0x410 [ 8220.462444] do_sys_open+0x73/0xa0 [ 8220.464155] __x64_sys_openat+0x24/0x30 [ 8220.466045] do_syscall_64+0xc1/0x440 [ 8220.473467] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8220.478348] RIP: 0033:0x7f1a63983332 [ 8220.483084] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8220.520133] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8220.531974] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8220.543081] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8220.557729] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8220.569981] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8220.579870] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8222.694713] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6199s! [ 8222.706257] Showing busy workqueues and worker pools: [ 8222.710044] workqueue events: flags=0x0 [ 8222.712340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8222.715608] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8222.724031] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8222.730772] pending: sec_gc_main [ptlrpc] [ 8222.740353] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8222.749404] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8222.767982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8222.778318] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8222.793376] workqueue events_long: flags=0x0 [ 8222.797033] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8222.803250] pending: lu_objects_flush [obdclass] [ 8222.807667] workqueue events_unbound: flags=0x2 [ 8222.810924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8222.814557] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8222.827902] workqueue events_power_efficient: flags=0x80 [ 8222.833119] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8222.837839] pending: neigh_periodic_work, do_cache_clean [ 8222.840723] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8222.844708] pending: neigh_periodic_work, check_lifetime [ 8222.847538] workqueue mm_percpu_wq: flags=0x8 [ 8222.859703] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8222.863331] pending: vmstat_update [ 8222.866891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8222.870693] pending: vmstat_update [ 8222.873220] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8222.875785] pending: vmstat_update [ 8222.879329] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8222.882174] pending: vmstat_update [ 8222.884622] workqueue writeback: flags=0x4a [ 8222.886502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8222.889324] pending: wb_workfn, wb_workfn [ 8222.891381] workqueue xprtiod: flags=0xa [ 8222.893178] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8222.895476] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8222.898550] workqueue ipv6_addrconf: flags=0x40008 [ 8222.900089] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8222.903774] pending: addrconf_verify_work [ 8222.905850] workqueue knbd0-recv: flags=0x1a [ 8222.908305] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8222.911317] in-flight: 647:recv_work [ 8222.913034] workqueue ptlrpc_pinger: flags=0x2 [ 8222.915469] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8222.918929] pending: ptlrpc_pinger_main [ptlrpc] [ 8222.921789] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8212s workers=2 idle: 652 [ 8228.056757] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8228.057353] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8228.066217] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8228.066232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8228.066310] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8228.066316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8228.066320] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8228.066334] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8228.066340] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8228.066345] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8228.066347] RDX: 0000000000003110 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8228.066350] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8228.066352] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8228.066355] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8228.066359] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8228.066362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8228.066365] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8228.066375] Call Trace: [ 8228.066381] [ 8228.066387] ? show_regs.cold.9+0x22/0x2f [ 8228.066398] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8228.066406] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8228.066411] ? watchdog+0x40/0x40 [ 8228.066416] ? hrtimer_interrupt+0x128/0x2c0 [ 8228.066421] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8228.066430] ? apic_timer_interrupt+0xf/0x20 [ 8228.066436] [ 8228.066439] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8228.066446] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8228.066452] do_raw_spin_lock+0x177/0x190 [ 8228.066457] _raw_spin_lock+0x21/0x30 [ 8228.066463] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8228.066790] obd_mod_free+0x46/0x140 [ptlrpc] [ 8228.067017] mdc_commit_open+0x91/0xb0 [mdc] [ 8228.067069] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8228.067274] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8228.067473] after_reply+0xe2c/0x1780 [ptlrpc] [ 8228.067667] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8228.067862] ? __next_timer_interrupt+0x160/0x160 [ 8228.067873] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8228.068075] ? do_wait_intr+0xf0/0xf0 [ 8228.068083] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8228.068288] kthread+0x1d1/0x200 [ 8228.068294] ? set_kthread_struct+0x70/0x70 [ 8228.068298] ret_from_fork+0x1f/0x30 [ 8228.072293] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8228.072377] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8228.072385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8228.072388] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8228.072404] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8228.072409] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8228.072414] RAX: 0000000000001f19 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8228.072416] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8228.072419] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8228.072421] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8228.072424] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8228.072427] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8228.072430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8228.072432] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8228.072442] Call Trace: [ 8228.072450] [ 8228.075319] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8228.075402] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8228.075410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8228.075412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8228.075428] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8228.075433] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8228.075438] RAX: 00000000000078a6 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8228.075440] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8228.075443] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8228.075445] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8228.075448] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8228.075451] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8228.075454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8228.075457] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8228.075465] Call Trace: [ 8228.075475] [ 8228.075482] ? show_regs.cold.9+0x22/0x2f [ 8228.075493] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8228.075499] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8228.075504] ? watchdog+0x40/0x40 [ 8228.075508] ? hrtimer_interrupt+0x128/0x2c0 [ 8228.075513] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8228.075522] ? apic_timer_interrupt+0xf/0x20 [ 8228.075526] [ 8228.075529] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8228.075538] do_raw_spin_lock+0x177/0x190 [ 8228.075544] _raw_spin_lock+0x21/0x30 [ 8228.075548] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8228.075967] ? do_raw_spin_unlock+0x75/0x190 [ 8228.075972] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8228.076184] ? do_wait_intr+0xf0/0xf0 [ 8228.076192] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8228.076637] mdc_close+0x53b/0xd70 [mdc] [ 8228.076680] lmv_close+0x18d/0x490 [lmv] [ 8228.076783] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8228.076896] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8228.076987] ll_md_close+0x274/0x800 [lustre] [ 8228.077069] ll_file_release+0x128/0xc50 [lustre] [ 8228.077161] __fput+0x10d/0x440 [ 8228.077170] ____fput+0x12/0x20 [ 8228.077175] task_work_run+0xc6/0x110 [ 8228.077185] exit_to_usermode_loop+0x1dd/0x1f0 [ 8228.077194] do_syscall_64+0x3d6/0x440 [ 8228.077199] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8228.077207] RIP: 0033:0x7f1a63983cd8 [ 8228.077764] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8228.077773] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8228.077779] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8228.077781] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8228.077783] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8228.077786] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8228.077788] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8229.013804] ? show_regs.cold.9+0x22/0x2f [ 8229.017680] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8229.021874] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8229.025925] ? watchdog+0x40/0x40 [ 8229.029634] ? hrtimer_interrupt+0x128/0x2c0 [ 8229.035218] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8229.041248] ? apic_timer_interrupt+0xf/0x20 [ 8229.042738] [ 8229.043508] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8229.055562] do_raw_spin_lock+0x177/0x190 [ 8229.058778] _raw_spin_lock+0x21/0x30 [ 8229.064326] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8229.071902] ? do_raw_spin_unlock+0x75/0x190 [ 8229.076676] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8229.085842] ? do_wait_intr+0xf0/0xf0 [ 8229.087149] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8229.095288] mdc_close+0x53b/0xd70 [mdc] [ 8229.098601] lmv_close+0x18d/0x490 [lmv] [ 8229.111025] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8229.114730] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8229.119818] ll_md_close+0x274/0x800 [lustre] [ 8229.129014] ll_file_release+0x128/0xc50 [lustre] [ 8229.130793] __fput+0x10d/0x440 [ 8229.139245] ____fput+0x12/0x20 [ 8229.140345] task_work_run+0xc6/0x110 [ 8229.141469] exit_to_usermode_loop+0x1dd/0x1f0 [ 8229.152810] do_syscall_64+0x3d6/0x440 [ 8229.158304] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8229.164355] RIP: 0033:0x7f1a63983cd8 [ 8229.167350] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8229.192075] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8229.201962] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8229.214442] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8229.231696] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8229.238731] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8229.242615] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8248.053025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8248.056317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8248.086976] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8248.093255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8248.096689] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8248.099431] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8248.109118] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8248.112107] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8248.114970] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8248.117735] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8248.121590] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8248.124464] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8248.127344] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8248.131955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8248.134778] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8248.138290] Call Trace: [ 8248.139466] [ 8248.140506] ? show_regs.cold.9+0x22/0x2f [ 8248.143870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8248.145491] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8248.147406] ? watchdog+0x40/0x40 [ 8248.148593] ? hrtimer_interrupt+0x128/0x2c0 [ 8248.150588] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8248.153314] ? apic_timer_interrupt+0xf/0x20 [ 8248.155318] [ 8248.160081] ? native_safe_halt+0x2e/0x50 [ 8248.161344] kvm_wait+0xcd/0xe0 [ 8248.162419] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8248.164433] do_raw_spin_lock+0x177/0x190 [ 8248.169218] _raw_spin_lock+0x21/0x30 [ 8248.173217] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8248.177786] ? do_raw_spin_unlock+0x75/0x190 [ 8248.179426] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8248.182135] ? do_wait_intr+0xf0/0xf0 [ 8248.183373] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8248.186263] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8248.191211] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8248.193652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8248.201273] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8248.204132] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8248.206368] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8248.211026] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8248.216050] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8248.221496] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8248.224091] lmv_intent_open+0x338/0x1150 [lmv] [ 8248.226472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8248.230614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8248.234203] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8248.236572] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8248.243133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8248.246814] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8248.254536] ? do_raw_spin_unlock+0x75/0x190 [ 8248.257764] ? _raw_spin_unlock+0x12/0x30 [ 8248.259605] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8248.263152] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8248.267713] ll_atomic_open+0x36a/0xd70 [lustre] [ 8248.270783] lookup_open+0x356/0xb40 [ 8248.273059] do_last+0x313/0xfc0 [ 8248.274767] ? nd_jump_root+0xe5/0x160 [ 8248.277071] ? path_init+0x437/0x520 [ 8248.279089] path_openat+0xf7/0x500 [ 8248.280600] ? do_raw_spin_unlock+0x75/0x190 [ 8248.282639] do_filp_open+0x99/0x140 [ 8248.284462] ? getname_flags+0x6e/0x330 [ 8248.286251] ? __check_object_size+0xff/0x256 [ 8248.288280] ? do_raw_spin_unlock+0x75/0x190 [ 8248.290816] ? _raw_spin_unlock+0x12/0x30 [ 8248.292350] do_sys_openat2+0x2b4/0x410 [ 8248.294628] do_sys_open+0x73/0xa0 [ 8248.295856] __x64_sys_openat+0x24/0x30 [ 8248.299064] do_syscall_64+0xc1/0x440 [ 8248.302840] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8248.312408] RIP: 0033:0x7f1a63983332 [ 8248.315892] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8248.333489] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8248.337847] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8248.341603] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8248.345179] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8248.348350] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8248.351636] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8253.410919] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6230s! [ 8253.419988] Showing busy workqueues and worker pools: [ 8253.424523] workqueue events: flags=0x0 [ 8253.427488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8253.434793] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8253.439946] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8253.442442] pending: sec_gc_main [ptlrpc] [ 8253.444993] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8253.447705] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8253.452828] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8253.456208] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8253.460830] workqueue events_long: flags=0x0 [ 8253.463018] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8253.465656] pending: lu_objects_flush [obdclass] [ 8253.468135] workqueue events_unbound: flags=0x2 [ 8253.470131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8253.473109] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8253.478130] workqueue events_power_efficient: flags=0x80 [ 8253.480845] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8253.484933] pending: neigh_periodic_work, do_cache_clean [ 8253.487316] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8253.489709] pending: neigh_periodic_work, check_lifetime [ 8253.492336] workqueue mm_percpu_wq: flags=0x8 [ 8253.494226] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8253.496606] pending: vmstat_update [ 8253.498330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8253.501067] pending: vmstat_update [ 8253.502853] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8253.505215] pending: vmstat_update [ 8253.506523] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8253.509381] pending: vmstat_update [ 8253.510852] workqueue writeback: flags=0x4a [ 8253.512453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8253.515459] pending: wb_workfn, wb_workfn [ 8253.517264] workqueue xprtiod: flags=0xa [ 8253.518779] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8253.521358] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8253.524638] workqueue ipv6_addrconf: flags=0x40008 [ 8253.526539] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8253.529264] pending: addrconf_verify_work [ 8253.531302] workqueue knbd0-recv: flags=0x1a [ 8253.532757] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8253.535509] in-flight: 647:recv_work [ 8253.537282] workqueue ptlrpc_pinger: flags=0x2 [ 8253.539233] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8253.541812] pending: ptlrpc_pinger_main [ptlrpc] [ 8253.544024] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8243s workers=2 idle: 652 [ 8256.056012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 8256.057004] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 8256.059795] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8256.059810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8256.059940] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8256.059946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8256.059948] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8256.059966] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8256.059970] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8256.059975] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8256.059977] RDX: 00000000000024b6 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8256.059979] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8256.059982] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8256.059984] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8256.059987] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8256.059990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8256.059992] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8256.060004] Call Trace: [ 8256.060011] [ 8256.060018] ? show_regs.cold.9+0x22/0x2f [ 8256.060028] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8256.060035] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8256.060046] ? watchdog+0x40/0x40 [ 8256.060051] ? hrtimer_interrupt+0x128/0x2c0 [ 8256.060055] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8256.060065] ? apic_timer_interrupt+0xf/0x20 [ 8256.060070] [ 8256.060077] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8256.060083] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8256.060088] do_raw_spin_lock+0x177/0x190 [ 8256.060098] _raw_spin_lock+0x21/0x30 [ 8256.060103] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8256.061061] obd_mod_free+0x46/0x140 [ptlrpc] [ 8256.061422] mdc_commit_open+0x91/0xb0 [mdc] [ 8256.061467] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8256.061774] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8256.062012] after_reply+0xe2c/0x1780 [ptlrpc] [ 8256.062307] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8256.062596] ? __next_timer_interrupt+0x160/0x160 [ 8256.062609] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8256.062950] ? do_wait_intr+0xf0/0xf0 [ 8256.062960] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8256.063186] kthread+0x1d1/0x200 [ 8256.063192] ? set_kthread_struct+0x70/0x70 [ 8256.063196] ret_from_fork+0x1f/0x30 [ 8256.065191] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8256.065274] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8256.065282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8256.065284] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8256.065299] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8256.065304] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8256.065308] RAX: 00000000000021d9 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8256.065310] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8256.065312] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8256.065314] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8256.065316] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8256.065319] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8256.065322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8256.065324] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8256.065332] Call Trace: [ 8256.065337] [ 8256.065342] ? show_regs.cold.9+0x22/0x2f [ 8256.065352] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8256.065359] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8256.065363] ? watchdog+0x40/0x40 [ 8256.065368] ? hrtimer_interrupt+0x128/0x2c0 [ 8256.065372] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8256.065380] ? apic_timer_interrupt+0xf/0x20 [ 8256.065398] [ 8256.065410] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8256.065417] do_raw_spin_lock+0x177/0x190 [ 8256.065421] _raw_spin_lock+0x21/0x30 [ 8256.065426] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8256.065713] ? do_raw_spin_unlock+0x75/0x190 [ 8256.065719] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8256.065931] ? do_wait_intr+0xf0/0xf0 [ 8256.065939] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8256.066133] mdc_close+0x53b/0xd70 [mdc] [ 8256.066172] lmv_close+0x18d/0x490 [lmv] [ 8256.066210] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8256.066332] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8256.066412] ll_md_close+0x274/0x800 [lustre] [ 8256.066492] ll_file_release+0x128/0xc50 [lustre] [ 8256.066572] __fput+0x10d/0x440 [ 8256.066583] ____fput+0x12/0x20 [ 8256.066587] task_work_run+0xc6/0x110 [ 8256.066594] exit_to_usermode_loop+0x1dd/0x1f0 [ 8256.066603] do_syscall_64+0x3d6/0x440 [ 8256.066608] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8256.066616] RIP: 0033:0x7f1a63983cd8 [ 8256.066623] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8256.066627] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8256.066632] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8256.066634] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8256.066636] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8256.066639] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8256.066641] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8256.591760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8256.628423] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8256.636824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8256.645570] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8256.656651] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8256.675608] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8256.680980] RAX: 0000000000005355 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8256.685453] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8256.690201] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8256.698445] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8256.703087] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8256.706184] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8256.711460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8256.713697] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8256.716315] Call Trace: [ 8256.717432] [ 8256.718488] ? show_regs.cold.9+0x22/0x2f [ 8256.721521] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8256.726147] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8256.728547] ? watchdog+0x40/0x40 [ 8256.731508] ? hrtimer_interrupt+0x128/0x2c0 [ 8256.733504] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8256.736598] ? apic_timer_interrupt+0xf/0x20 [ 8256.739011] [ 8256.739809] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8256.742505] do_raw_spin_lock+0x177/0x190 [ 8256.744138] _raw_spin_lock+0x21/0x30 [ 8256.745814] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8256.748781] ? do_raw_spin_unlock+0x75/0x190 [ 8256.753355] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8256.759199] ? do_wait_intr+0xf0/0xf0 [ 8256.760745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8256.763039] mdc_close+0x53b/0xd70 [mdc] [ 8256.764326] lmv_close+0x18d/0x490 [lmv] [ 8256.768594] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8256.770396] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8256.775528] ll_md_close+0x274/0x800 [lustre] [ 8256.777266] ll_file_release+0x128/0xc50 [lustre] [ 8256.780748] __fput+0x10d/0x440 [ 8256.783042] ____fput+0x12/0x20 [ 8256.785327] task_work_run+0xc6/0x110 [ 8256.788453] exit_to_usermode_loop+0x1dd/0x1f0 [ 8256.791672] do_syscall_64+0x3d6/0x440 [ 8256.793547] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8256.797357] RIP: 0033:0x7f1a63983cd8 [ 8256.798416] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8256.811149] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8256.813562] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8256.821209] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8256.823413] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8256.833173] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8256.842288] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8280.054763] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8280.059383] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8280.104195] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8280.111722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8280.123875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8280.128371] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8280.146579] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8280.152948] RAX: 0000000000007c86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8280.159111] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8280.167464] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8280.171402] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8280.176870] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8280.181343] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8280.185776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8280.189086] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8280.195716] Call Trace: [ 8280.198901] [ 8280.200668] ? show_regs.cold.9+0x22/0x2f [ 8280.204156] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8280.206399] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8280.209850] ? watchdog+0x40/0x40 [ 8280.211945] ? hrtimer_interrupt+0x128/0x2c0 [ 8280.217943] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8280.220279] ? apic_timer_interrupt+0xf/0x20 [ 8280.222946] [ 8280.224147] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8280.226329] do_raw_spin_lock+0x177/0x190 [ 8280.227737] _raw_spin_lock+0x21/0x30 [ 8280.229765] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8280.232436] ? do_raw_spin_unlock+0x75/0x190 [ 8280.234421] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8280.237709] ? do_wait_intr+0xf0/0xf0 [ 8280.239844] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8280.242735] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8280.245295] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8280.248353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8280.250931] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8280.253718] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8280.256433] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8280.259039] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8280.261824] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8280.265339] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8280.268946] lmv_intent_open+0x338/0x1150 [lmv] [ 8280.270846] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8280.274171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8280.277354] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8280.280201] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8280.283134] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8280.285986] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8280.288791] ? do_raw_spin_unlock+0x75/0x190 [ 8280.290579] ? _raw_spin_unlock+0x12/0x30 [ 8280.291927] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8280.293970] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8280.296322] ll_atomic_open+0x36a/0xd70 [lustre] [ 8280.299087] lookup_open+0x356/0xb40 [ 8280.300782] do_last+0x313/0xfc0 [ 8280.302328] ? nd_jump_root+0xe5/0x160 [ 8280.304776] ? path_init+0x437/0x520 [ 8280.306445] path_openat+0xf7/0x500 [ 8280.309872] ? do_raw_spin_unlock+0x75/0x190 [ 8280.312913] do_filp_open+0x99/0x140 [ 8280.315021] ? getname_flags+0x6e/0x330 [ 8280.317971] ? __check_object_size+0xff/0x256 [ 8280.321008] ? do_raw_spin_unlock+0x75/0x190 [ 8280.323212] ? _raw_spin_unlock+0x12/0x30 [ 8280.325659] do_sys_openat2+0x2b4/0x410 [ 8280.329710] do_sys_open+0x73/0xa0 [ 8280.331377] __x64_sys_openat+0x24/0x30 [ 8280.333116] do_syscall_64+0xc1/0x440 [ 8280.335392] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8280.338659] RIP: 0033:0x7f1a63983332 [ 8280.339842] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8280.350191] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8280.353436] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8280.356832] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8280.360826] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8280.366308] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8280.370006] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8283.616046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6260s! [ 8283.626679] Showing busy workqueues and worker pools: [ 8283.630253] workqueue events: flags=0x0 [ 8283.632435] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8283.637416] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8283.646773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8283.652706] pending: sec_gc_main [ptlrpc] [ 8283.655874] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8283.661151] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8283.673083] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8283.678778] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8283.684244] workqueue events_long: flags=0x0 [ 8283.692262] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8283.698884] pending: lu_objects_flush [obdclass] [ 8283.709174] workqueue events_unbound: flags=0x2 [ 8283.715156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8283.727886] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8283.733438] workqueue events_power_efficient: flags=0x80 [ 8283.736301] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8283.739433] pending: neigh_periodic_work, do_cache_clean [ 8283.742217] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8283.745696] pending: neigh_periodic_work, check_lifetime [ 8283.748336] workqueue mm_percpu_wq: flags=0x8 [ 8283.751134] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8283.758245] pending: vmstat_update [ 8283.763012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8283.771764] pending: vmstat_update [ 8283.774273] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8283.779325] pending: vmstat_update [ 8283.782723] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8283.787901] pending: vmstat_update [ 8283.791784] workqueue writeback: flags=0x4a [ 8283.795750] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8283.800322] pending: wb_workfn, wb_workfn [ 8283.804536] workqueue xprtiod: flags=0xa [ 8283.808390] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8283.815523] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8283.820188] workqueue ipv6_addrconf: flags=0x40008 [ 8283.823712] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8283.831680] pending: addrconf_verify_work [ 8283.835750] workqueue knbd0-recv: flags=0x1a [ 8283.838301] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8283.845416] in-flight: 647:recv_work [ 8283.847503] workqueue ptlrpc_pinger: flags=0x2 [ 8283.849459] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8283.856762] pending: ptlrpc_pinger_main [ptlrpc] [ 8283.864663] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8273s workers=2 idle: 652 [ 8284.056027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 8284.057054] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 8284.059429] Modules linked in: [ 8284.061438] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8284.061450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8284.061523] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8284.061528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8284.061531] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8284.061543] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8284.061548] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8284.061552] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8284.061555] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8284.061557] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8284.061559] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8284.061562] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8284.061565] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8284.061568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8284.061570] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8284.061587] Call Trace: [ 8284.061593] [ 8284.061598] ? show_regs.cold.9+0x22/0x2f [ 8284.061607] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8284.061613] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8284.061617] ? watchdog+0x40/0x40 [ 8284.061622] ? hrtimer_interrupt+0x128/0x2c0 [ 8284.061626] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8284.061632] ? apic_timer_interrupt+0xf/0x20 [ 8284.061637] [ 8284.061639] ? native_safe_halt+0x2e/0x50 [ 8284.061644] kvm_wait+0xcd/0xe0 [ 8284.061651] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8284.061659] do_raw_spin_lock+0x177/0x190 [ 8284.061663] _raw_spin_lock+0x21/0x30 [ 8284.061668] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8284.061955] obd_mod_free+0x46/0x140 [ptlrpc] [ 8284.062160] mdc_commit_open+0x91/0xb0 [mdc] [ 8284.062197] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8284.062399] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8284.062672] after_reply+0xe2c/0x1780 [ptlrpc] [ 8284.063012] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8284.063214] ? __next_timer_interrupt+0x160/0x160 [ 8284.063224] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8284.063435] ? do_wait_intr+0xf0/0xf0 [ 8284.063444] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8284.063665] kthread+0x1d1/0x200 [ 8284.063670] ? set_kthread_struct+0x70/0x70 [ 8284.063675] ret_from_fork+0x1f/0x30 [ 8284.067817] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [ 8284.068841] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8284.068889] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8284.068893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8284.068896] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8284.068906] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8284.068909] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8284.068912] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8284.068914] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8284.068916] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8284.068917] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8284.068919] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8284.068921] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8284.068923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8284.068925] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8284.068930] Call Trace: [ 8284.068935] [ 8284.068939] ? show_regs.cold.9+0x22/0x2f [ 8284.068947] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8284.068951] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8284.068954] ? watchdog+0x40/0x40 [ 8284.068956] ? hrtimer_interrupt+0x128/0x2c0 [ 8284.068959] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8284.068964] ? apic_timer_interrupt+0xf/0x20 [ 8284.068967] [ 8284.068969] ? native_safe_halt+0x2e/0x50 [ 8284.068972] kvm_wait+0xcd/0xe0 [ 8284.068978] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8284.068983] do_raw_spin_lock+0x177/0x190 [ 8284.068986] _raw_spin_lock+0x21/0x30 [ 8284.068989] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8284.071945] dns_resolver [ 8284.085480] ? do_raw_spin_unlock+0x75/0x190 [ 8284.092415] intel_rapl_msr [ 8284.095929] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8284.096231] ? do_wait_intr+0xf0/0xf0 [ 8284.096242] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8284.096446] mdc_close+0x53b/0xd70 [mdc] [ 8284.096484] lmv_close+0x18d/0x490 [lmv] [ 8284.096516] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8284.096868] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8284.096953] ll_md_close+0x274/0x800 [lustre] [ 8284.097037] ll_file_release+0x128/0xc50 [lustre] [ 8284.097121] __fput+0x10d/0x440 [ 8284.097132] ____fput+0x12/0x20 [ 8284.097136] task_work_run+0xc6/0x110 [ 8284.097143] exit_to_usermode_loop+0x1dd/0x1f0 [ 8284.097150] do_syscall_64+0x3d6/0x440 [ 8284.097155] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8284.097164] RIP: 0033:0x7f1a63983cd8 [ 8284.097172] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8284.097176] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8284.097182] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8284.097184] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8284.097186] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8284.097189] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8284.097191] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8284.462235] intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8284.473377] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8284.479304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8284.483938] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8284.488720] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8284.500267] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8284.505484] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8284.509541] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8284.513157] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8284.517664] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8284.521139] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8284.527584] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8284.533473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8284.540328] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8284.543333] Call Trace: [ 8284.544352] [ 8284.545010] ? show_regs.cold.9+0x22/0x2f [ 8284.546776] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8284.549378] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8284.552780] ? watchdog+0x40/0x40 [ 8284.554416] ? hrtimer_interrupt+0x128/0x2c0 [ 8284.556253] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8284.562003] ? apic_timer_interrupt+0xf/0x20 [ 8284.566890] [ 8284.567652] ? native_safe_halt+0x2e/0x50 [ 8284.569087] kvm_wait+0xcd/0xe0 [ 8284.570289] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8284.575948] do_raw_spin_lock+0x177/0x190 [ 8284.578959] _raw_spin_lock+0x21/0x30 [ 8284.583177] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8284.586392] ? do_raw_spin_unlock+0x75/0x190 [ 8284.588036] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8284.592110] ? do_wait_intr+0xf0/0xf0 [ 8284.594245] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8284.596813] mdc_close+0x53b/0xd70 [mdc] [ 8284.598878] lmv_close+0x18d/0x490 [lmv] [ 8284.602858] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8284.611364] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8284.613273] ll_md_close+0x274/0x800 [lustre] [ 8284.620311] ll_file_release+0x128/0xc50 [lustre] [ 8284.623807] __fput+0x10d/0x440 [ 8284.628323] ____fput+0x12/0x20 [ 8284.630492] task_work_run+0xc6/0x110 [ 8284.635248] exit_to_usermode_loop+0x1dd/0x1f0 [ 8284.636502] do_syscall_64+0x3d6/0x440 [ 8284.641127] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8284.643765] RIP: 0033:0x7f1a63983cd8 [ 8284.647290] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8284.665976] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8284.678231] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8284.681166] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8284.689405] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8284.694126] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8284.706027] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8308.053010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8308.056777] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8308.079934] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8308.084239] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8308.087222] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8308.089283] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8308.094733] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8308.097142] RAX: 00000000000018d6 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8308.099302] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8308.101797] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8308.104178] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8308.106850] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8308.109218] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8308.111808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8308.113538] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8308.116430] Call Trace: [ 8308.117307] [ 8308.117929] ? show_regs.cold.9+0x22/0x2f [ 8308.119028] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8308.120759] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8308.122099] ? watchdog+0x40/0x40 [ 8308.124029] ? hrtimer_interrupt+0x128/0x2c0 [ 8308.128869] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8308.132145] ? apic_timer_interrupt+0xf/0x20 [ 8308.136919] [ 8308.137780] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8308.139766] do_raw_spin_lock+0x177/0x190 [ 8308.141316] _raw_spin_lock+0x21/0x30 [ 8308.142788] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8308.144823] ? do_raw_spin_unlock+0x75/0x190 [ 8308.146168] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8308.147938] ? do_wait_intr+0xf0/0xf0 [ 8308.149151] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8308.151580] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8308.155224] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8308.158109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8308.161556] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8308.163496] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8308.165481] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8308.167283] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8308.170209] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8308.173477] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8308.181499] lmv_intent_open+0x338/0x1150 [lmv] [ 8308.184460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8308.188376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8308.190507] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8308.193070] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8308.195073] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8308.197330] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8308.199663] ? do_raw_spin_unlock+0x75/0x190 [ 8308.201454] ? _raw_spin_unlock+0x12/0x30 [ 8308.202887] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8308.204948] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8308.206908] ll_atomic_open+0x36a/0xd70 [lustre] [ 8308.208608] lookup_open+0x356/0xb40 [ 8308.209722] do_last+0x313/0xfc0 [ 8308.210532] ? nd_jump_root+0xe5/0x160 [ 8308.211713] ? path_init+0x437/0x520 [ 8308.212757] path_openat+0xf7/0x500 [ 8308.213869] ? do_raw_spin_unlock+0x75/0x190 [ 8308.215243] do_filp_open+0x99/0x140 [ 8308.216712] ? getname_flags+0x6e/0x330 [ 8308.218478] ? __check_object_size+0xff/0x256 [ 8308.220095] ? do_raw_spin_unlock+0x75/0x190 [ 8308.221545] ? _raw_spin_unlock+0x12/0x30 [ 8308.227330] do_sys_openat2+0x2b4/0x410 [ 8308.228324] do_sys_open+0x73/0xa0 [ 8308.229223] __x64_sys_openat+0x24/0x30 [ 8308.238086] do_syscall_64+0xc1/0x440 [ 8308.240850] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8308.243016] RIP: 0033:0x7f1a63983332 [ 8308.248026] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8308.264396] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8308.271903] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8308.275813] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8308.279539] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8308.282871] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8308.285787] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8312.056033] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 8312.058129] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8312.059088] Modules linked in: [ 8312.060129] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 8312.060138] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8312.060216] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8312.060221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8312.060224] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8312.060237] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8312.060241] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8312.060246] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8312.060248] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8312.060251] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8312.060253] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8312.060255] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8312.060258] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8312.060261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8312.060264] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8312.060274] Call Trace: [ 8312.060279] [ 8312.060285] ? show_regs.cold.9+0x22/0x2f [ 8312.060294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8312.060311] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8312.060316] ? watchdog+0x40/0x40 [ 8312.060321] ? hrtimer_interrupt+0x128/0x2c0 [ 8312.060325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8312.060332] ? apic_timer_interrupt+0xf/0x20 [ 8312.060341] [ 8312.060344] ? native_safe_halt+0x2e/0x50 [ 8312.060347] kvm_wait+0xcd/0xe0 [ 8312.060354] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8312.060363] do_raw_spin_lock+0x177/0x190 [ 8312.060368] _raw_spin_lock+0x21/0x30 [ 8312.060373] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8312.060665] ? do_raw_spin_unlock+0x75/0x190 [ 8312.060670] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8312.060795] ? do_wait_intr+0xf0/0xf0 [ 8312.060801] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8312.060979] mdc_close+0x53b/0xd70 [mdc] [ 8312.061020] lmv_close+0x18d/0x490 [lmv] [ 8312.061049] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8312.061156] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8312.061237] ll_md_close+0x274/0x800 [lustre] [ 8312.061334] ll_file_release+0x128/0xc50 [lustre] [ 8312.061429] __fput+0x10d/0x440 [ 8312.061437] ____fput+0x12/0x20 [ 8312.061440] task_work_run+0xc6/0x110 [ 8312.061445] exit_to_usermode_loop+0x1dd/0x1f0 [ 8312.061450] do_syscall_64+0x3d6/0x440 [ 8312.061453] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8312.061457] RIP: 0033:0x7f1a63983cd8 [ 8312.061462] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8312.061466] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8312.061470] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8312.061472] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8312.061475] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8312.061477] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8312.061479] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8312.062516] Modules linked in: [ 8312.063468] lustre(O) [ 8312.067781] lustre(O) [ 8312.096823] osp(O) ofd(O) lod(O) [ 8312.105447] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8312.105528] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8312.105537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8312.105540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8312.105656] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8312.105661] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8312.105666] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8312.105669] RDX: 0000000000003e13 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8312.105672] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8312.105674] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8312.105676] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8312.105679] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8312.105682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8312.105685] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8312.105693] Call Trace: [ 8312.105698] [ 8312.105709] ? show_regs.cold.9+0x22/0x2f [ 8312.105719] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8312.105731] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8312.105736] ? watchdog+0x40/0x40 [ 8312.105741] ? hrtimer_interrupt+0x128/0x2c0 [ 8312.105746] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8312.105754] ? apic_timer_interrupt+0xf/0x20 [ 8312.105758] [ 8312.105761] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8312.105766] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8312.105770] do_raw_spin_lock+0x177/0x190 [ 8312.105775] _raw_spin_lock+0x21/0x30 [ 8312.105784] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8312.111836] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8312.112007] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8312.112014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8312.112018] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8312.112030] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8312.112035] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8312.112040] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8312.112042] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8312.112044] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8312.112046] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8312.112048] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8312.112051] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8312.112053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8312.112055] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8312.112067] Call Trace: [ 8312.112073] [ 8312.112079] ? show_regs.cold.9+0x22/0x2f [ 8312.112089] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8312.112095] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8312.112100] ? watchdog+0x40/0x40 [ 8312.112104] ? hrtimer_interrupt+0x128/0x2c0 [ 8312.112108] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8312.112115] ? apic_timer_interrupt+0xf/0x20 [ 8312.112119] [ 8312.112122] ? native_safe_halt+0x2e/0x50 [ 8312.112126] kvm_wait+0xcd/0xe0 [ 8312.112133] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8312.112142] do_raw_spin_lock+0x177/0x190 [ 8312.112146] _raw_spin_lock+0x21/0x30 [ 8312.112151] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8312.112577] ? do_raw_spin_unlock+0x75/0x190 [ 8312.112584] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8312.112911] ? do_wait_intr+0xf0/0xf0 [ 8312.112920] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8312.113134] mdc_close+0x53b/0xd70 [mdc] [ 8312.113173] lmv_close+0x18d/0x490 [lmv] [ 8312.113210] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8312.113403] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8312.113560] ll_md_close+0x274/0x800 [lustre] [ 8312.113642] ll_file_release+0x128/0xc50 [lustre] [ 8312.113730] __fput+0x10d/0x440 [ 8312.113739] ____fput+0x12/0x20 [ 8312.113743] task_work_run+0xc6/0x110 [ 8312.113751] exit_to_usermode_loop+0x1dd/0x1f0 [ 8312.113758] do_syscall_64+0x3d6/0x440 [ 8312.113763] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8312.113768] RIP: 0033:0x7f1a63983cd8 [ 8312.113774] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8312.113778] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8312.113783] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8312.113785] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8312.113787] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8312.113789] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8312.113791] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8312.604732] obd_mod_free+0x46/0x140 [ptlrpc] [ 8312.608486] mdc_commit_open+0x91/0xb0 [mdc] [ 8312.613263] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8312.623762] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8312.627739] after_reply+0xe2c/0x1780 [ptlrpc] [ 8312.630928] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8312.634588] ? __next_timer_interrupt+0x160/0x160 [ 8312.637982] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8312.642226] ? do_wait_intr+0xf0/0xf0 [ 8312.647189] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8312.653172] kthread+0x1d1/0x200 [ 8312.657484] ? set_kthread_struct+0x70/0x70 [ 8312.661170] ret_from_fork+0x1f/0x30 [ 8314.337158] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6291s! [ 8314.339877] Showing busy workqueues and worker pools: [ 8314.345748] workqueue events: flags=0x0 [ 8314.349785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8314.353601] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8314.361300] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8314.365044] pending: sec_gc_main [ptlrpc] [ 8314.369586] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8314.376193] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8314.390094] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8314.400219] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8314.418864] workqueue events_long: flags=0x0 [ 8314.422062] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8314.430634] pending: lu_objects_flush [obdclass] [ 8314.437029] workqueue events_unbound: flags=0x2 [ 8314.441361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8314.443751] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8314.461016] workqueue events_power_efficient: flags=0x80 [ 8314.468124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8314.473906] pending: neigh_periodic_work, do_cache_clean [ 8314.480899] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8314.487493] pending: neigh_periodic_work, check_lifetime [ 8314.491126] workqueue mm_percpu_wq: flags=0x8 [ 8314.499075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8314.504858] pending: vmstat_update [ 8314.511227] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8314.520800] pending: vmstat_update [ 8314.525370] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8314.529802] pending: vmstat_update [ 8314.538474] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8314.541186] pending: vmstat_update [ 8314.548878] workqueue writeback: flags=0x4a [ 8314.552902] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8314.558229] pending: wb_workfn, wb_workfn [ 8314.564024] workqueue xprtiod: flags=0xa [ 8314.569994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8314.574499] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8314.580703] workqueue ipv6_addrconf: flags=0x40008 [ 8314.586843] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8314.589493] pending: addrconf_verify_work [ 8314.599058] workqueue knbd0-recv: flags=0x1a [ 8314.607823] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8314.616946] in-flight: 647:recv_work [ 8314.620255] workqueue ptlrpc_pinger: flags=0x2 [ 8314.625363] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8314.629049] pending: ptlrpc_pinger_main [ptlrpc] [ 8314.636900] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8304s workers=2 idle: 652 [ 8340.053048] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 8340.059040] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 8340.059055] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8340.059056] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 8340.059066] Modules linked in: [ 8340.059067] Modules linked in: [ 8340.059070] lustre(O) osp(O) [ 8340.059072] lustre(O) [ 8340.059079] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr [ 8340.059087] osp(O) [ 8340.059131] i2c_piix4 [ 8340.059135] ofd(O) [ 8340.059136] squashfs [ 8340.059142] lod(O) [ 8340.059142] crct10dif_pclmul [ 8340.059147] mdt(O) [ 8340.059148] crc32_pclmul [ 8340.059153] mdd(O) [ 8340.059154] crc32c_intel [ 8340.059159] mgs(O) [ 8340.059160] ata_generic ghash_clmulni_intel [ 8340.059166] lquota(O) [ 8340.059170] ata_piix [ 8340.059172] lfsck(O) [ 8340.059174] serio_raw [ 8340.059178] obdecho(O) [ 8340.059180] libata [ 8340.059184] mgc(O) [ 8340.059186] dm_mirror [ 8340.059190] mdc(O) [ 8340.059191] dm_region_hash [ 8340.059195] lov(O) [ 8340.059197] dm_log [ 8340.059201] osc(O) [ 8340.059202] dm_mod [ 8340.059207] lmv(O) [ 8340.059208] sha512_ssse3 sha512_generic [ 8340.059214] ec(O) [ 8340.059219] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8340.059223] fid(O) [ 8340.059229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8340.059238] fld(O) [ 8340.059233] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8340.059243] ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 8340.059250] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8340.059254] ksocklnd(O) [ 8340.059260] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 [ 8340.059261] lnet(O) libcfs(O) [ 8340.059267] ORIG_RAX: ffffffffffffff13 [ 8340.059269] rpcsec_gss_krb5 [ 8340.059273] RAX: 0000000000000d3f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8340.059274] auth_rpcgss [ 8340.059279] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8340.059280] nfsv4 dns_resolver [ 8340.059286] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8340.059290] intel_rapl_msr [ 8340.059293] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8340.059295] intel_rapl_common [ 8340.059299] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8340.059301] sb_edac rapl pcspkr [ 8340.059307] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8340.059311] i2c_piix4 squashfs [ 8340.059316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8340.059322] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8340.059325] crct10dif_pclmul crc32_pclmul [ 8340.059334] Call Trace: [ 8340.059335] crc32c_intel ata_generic ghash_clmulni_intel [ 8340.059343] [ 8340.059344] ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [ 8340.059352] ? show_regs.cold.9+0x22/0x2f [ 8340.059361] sha512_generic [ 8340.059369] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8340.059368] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8340.059378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8340.059381] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8340.059389] ? watchdog+0x40/0x40 [ 8340.059383] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8340.059396] ? hrtimer_interrupt+0x128/0x2c0 [ 8340.059402] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8340.059411] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 8340.059405] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8340.059420] ORIG_RAX: ffffffffffffff13 [ 8340.059420] ? apic_timer_interrupt+0xf/0x20 [ 8340.059428] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8340.059432] RDX: 000000000000618f RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8340.059435] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8340.059437] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8340.059443] [ 8340.059443] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8340.059450] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8340.059454] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8340.059462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8340.059461] do_raw_spin_lock+0x177/0x190 [ 8340.059467] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8340.059471] _raw_spin_lock+0x21/0x30 [ 8340.059477] Call Trace: [ 8340.059484] [ 8340.059489] ? show_regs.cold.9+0x22/0x2f [ 8340.059497] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8340.059503] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8340.059511] Modules linked in: [ 8340.059522] ? watchdog+0x40/0x40 [ 8340.059528] ? hrtimer_interrupt+0x128/0x2c0 [ 8340.059532] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8340.059539] ? apic_timer_interrupt+0xf/0x20 [ 8340.059544] [ 8340.059548] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8340.059553] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8340.059558] do_raw_spin_lock+0x177/0x190 [ 8340.059564] _raw_spin_lock+0x21/0x30 [ 8340.059569] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8340.059958] obd_mod_free+0x46/0x140 [ptlrpc] [ 8340.060159] mdc_commit_open+0x91/0xb0 [mdc] [ 8340.060192] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8340.059479] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8340.060403] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8340.061022] ? do_raw_spin_unlock+0x75/0x190 [ 8340.061041] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8340.061173] after_reply+0xe2c/0x1780 [ptlrpc] [ 8340.061702] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8340.061948] ? __next_timer_interrupt+0x160/0x160 [ 8340.061963] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8340.062203] ? do_wait_intr+0xf0/0xf0 [ 8340.062675] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8340.062901] kthread+0x1d1/0x200 [ 8340.062906] ? set_kthread_struct+0x70/0x70 [ 8340.062910] ret_from_fork+0x1f/0x30 [ 8340.063274] ? do_wait_intr+0xf0/0xf0 [ 8340.063282] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8340.063515] mdc_close+0x53b/0xd70 [mdc] [ 8340.064087] lmv_close+0x18d/0x490 [lmv] [ 8340.064118] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8340.064232] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8340.064317] ll_md_close+0x274/0x800 [lustre] [ 8340.064399] ll_file_release+0x128/0xc50 [lustre] [ 8340.064478] __fput+0x10d/0x440 [ 8340.064485] ____fput+0x12/0x20 [ 8340.064489] task_work_run+0xc6/0x110 [ 8340.064496] exit_to_usermode_loop+0x1dd/0x1f0 [ 8340.064504] do_syscall_64+0x3d6/0x440 [ 8340.064508] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8340.064515] RIP: 0033:0x7f1a63983cd8 [ 8340.064522] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8340.065515] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8340.065520] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8340.065523] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8340.065525] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8340.065528] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8340.065530] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8340.076949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8340.077044] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8340.077051] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8340.077054] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8340.077066] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8340.077071] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8340.077076] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8340.077079] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8340.077081] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8340.077083] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8340.077086] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8340.077089] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8340.077092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8340.077095] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8340.077105] Call Trace: [ 8340.077111] [ 8340.077117] ? show_regs.cold.9+0x22/0x2f [ 8340.077127] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8340.077146] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8340.077150] ? watchdog+0x40/0x40 [ 8340.077153] ? hrtimer_interrupt+0x128/0x2c0 [ 8340.077157] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8340.077166] ? apic_timer_interrupt+0xf/0x20 [ 8340.077170] [ 8340.077174] ? native_safe_halt+0x2e/0x50 [ 8340.077177] kvm_wait+0xcd/0xe0 [ 8340.077184] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8340.077192] do_raw_spin_lock+0x177/0x190 [ 8340.077197] _raw_spin_lock+0x21/0x30 [ 8340.077202] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8340.077494] ? do_raw_spin_unlock+0x75/0x190 [ 8340.077500] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8340.084168] lustre(O) [ 8340.098070] ? do_wait_intr+0xf0/0xf0 [ 8340.098090] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8340.098720] mdc_close+0x53b/0xd70 [mdc] [ 8340.098760] lmv_close+0x18d/0x490 [lmv] [ 8340.098793] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8340.098920] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8340.099008] ll_md_close+0x274/0x800 [lustre] [ 8340.099100] ll_file_release+0x128/0xc50 [lustre] [ 8340.099437] __fput+0x10d/0x440 [ 8340.099446] ____fput+0x12/0x20 [ 8340.099451] task_work_run+0xc6/0x110 [ 8340.099458] exit_to_usermode_loop+0x1dd/0x1f0 [ 8340.099465] do_syscall_64+0x3d6/0x440 [ 8340.099470] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8340.099479] RIP: 0033:0x7f1a63983cd8 [ 8340.099486] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8340.099490] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8340.099494] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8340.099496] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8340.099498] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8340.099501] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8340.099503] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8341.108627] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8341.165276] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8341.172437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8341.178347] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8341.184556] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8341.197250] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8341.200912] RAX: 0000000000003055 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8341.206775] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8341.215419] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8341.224815] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8341.241949] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8341.250680] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8341.258773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8341.261985] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8341.267874] Call Trace: [ 8341.270900] [ 8341.272311] ? show_regs.cold.9+0x22/0x2f [ 8341.273691] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8341.275654] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8341.277277] ? watchdog+0x40/0x40 [ 8341.278565] ? hrtimer_interrupt+0x128/0x2c0 [ 8341.281617] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8341.284116] ? apic_timer_interrupt+0xf/0x20 [ 8341.286289] [ 8341.287295] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8341.290201] do_raw_spin_lock+0x177/0x190 [ 8341.292289] _raw_spin_lock+0x21/0x30 [ 8341.294080] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8341.299811] ? do_raw_spin_unlock+0x75/0x190 [ 8341.303944] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8341.311392] ? do_wait_intr+0xf0/0xf0 [ 8341.314743] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8341.318312] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8341.321652] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8341.324745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8341.327577] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8341.330305] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8341.332682] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8341.337804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8341.346413] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8341.349027] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8341.358526] lmv_intent_open+0x338/0x1150 [lmv] [ 8341.367737] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8341.370527] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8341.375523] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8341.376705] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8341.391460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8341.393478] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8341.398634] ? do_raw_spin_unlock+0x75/0x190 [ 8341.402628] ? _raw_spin_unlock+0x12/0x30 [ 8341.406913] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8341.410750] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8341.413803] ll_atomic_open+0x36a/0xd70 [lustre] [ 8341.424627] lookup_open+0x356/0xb40 [ 8341.427222] do_last+0x313/0xfc0 [ 8341.430320] ? nd_jump_root+0xe5/0x160 [ 8341.432222] ? path_init+0x437/0x520 [ 8341.437328] path_openat+0xf7/0x500 [ 8341.439678] ? do_raw_spin_unlock+0x75/0x190 [ 8341.441413] do_filp_open+0x99/0x140 [ 8341.442781] ? getname_flags+0x6e/0x330 [ 8341.445270] ? __check_object_size+0xff/0x256 [ 8341.448508] ? do_raw_spin_unlock+0x75/0x190 [ 8341.453961] ? _raw_spin_unlock+0x12/0x30 [ 8341.459679] do_sys_openat2+0x2b4/0x410 [ 8341.462169] do_sys_open+0x73/0xa0 [ 8341.464694] __x64_sys_openat+0x24/0x30 [ 8341.467484] do_syscall_64+0xc1/0x440 [ 8341.469835] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8341.473808] RIP: 0033:0x7f1a63983332 [ 8341.475770] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8341.493993] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8341.502799] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8341.510573] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8341.517285] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8341.521219] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8341.523732] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8345.057129] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6322s! [ 8345.061334] Showing busy workqueues and worker pools: [ 8345.068630] workqueue events: flags=0x0 [ 8345.071517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8345.076930] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8345.082334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8345.088866] pending: sec_gc_main [ptlrpc] [ 8345.091773] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8345.098623] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8345.107664] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8345.115052] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8345.123390] workqueue events_long: flags=0x0 [ 8345.124878] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8345.131943] pending: lu_objects_flush [obdclass] [ 8345.135631] workqueue events_unbound: flags=0x2 [ 8345.137659] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8345.140586] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8345.144402] workqueue events_power_efficient: flags=0x80 [ 8345.146400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8345.148917] pending: neigh_periodic_work, do_cache_clean [ 8345.151029] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8345.153676] pending: neigh_periodic_work, check_lifetime [ 8345.155575] workqueue mm_percpu_wq: flags=0x8 [ 8345.156835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8345.159187] pending: vmstat_update [ 8345.160446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8345.162817] pending: vmstat_update [ 8345.164011] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8345.166365] pending: vmstat_update [ 8345.167701] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8345.170055] pending: vmstat_update [ 8345.171323] workqueue writeback: flags=0x4a [ 8345.172609] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8345.174989] pending: wb_workfn, wb_workfn [ 8345.176481] workqueue xprtiod: flags=0xa [ 8345.178049] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8345.180136] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8345.183355] workqueue ipv6_addrconf: flags=0x40008 [ 8345.185106] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8345.187620] pending: addrconf_verify_work [ 8345.189443] workqueue knbd0-recv: flags=0x1a [ 8345.191364] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8345.193807] in-flight: 647:recv_work [ 8345.195355] workqueue ptlrpc_pinger: flags=0x2 [ 8345.196707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8345.198971] pending: ptlrpc_pinger_main [ptlrpc] [ 8345.201472] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8335s workers=2 idle: 652 [ 8368.056115] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8368.057142] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8368.058030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8368.058038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8368.058103] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8368.058109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8368.058112] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8368.058125] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8368.058129] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8368.058133] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8368.058136] RDX: 0000000000002fbd RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8368.058138] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8368.058140] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8368.058142] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8368.058145] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8368.058148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8368.058150] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8368.058159] Call Trace: [ 8368.058164] [ 8368.058236] ? show_regs.cold.9+0x22/0x2f [ 8368.058246] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8368.058252] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8368.058260] ? watchdog+0x40/0x40 [ 8368.058264] ? hrtimer_interrupt+0x128/0x2c0 [ 8368.058268] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8368.058276] ? apic_timer_interrupt+0xf/0x20 [ 8368.058280] [ 8368.058283] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8368.058288] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8368.058292] do_raw_spin_lock+0x177/0x190 [ 8368.058297] _raw_spin_lock+0x21/0x30 [ 8368.058301] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8368.058980] obd_mod_free+0x46/0x140 [ptlrpc] [ 8368.059304] mdc_commit_open+0x91/0xb0 [mdc] [ 8368.059345] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8368.059815] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8368.060026] after_reply+0xe2c/0x1780 [ptlrpc] [ 8368.060388] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8368.060977] ? __next_timer_interrupt+0x160/0x160 [ 8368.060988] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8368.061216] ? do_wait_intr+0xf0/0xf0 [ 8368.061231] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8368.061402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8368.061474] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8368.061480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8368.061483] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8368.061495] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8368.061499] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8368.061505] RAX: 0000000000007786 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8368.061507] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8368.061509] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8368.061512] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8368.061514] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8368.061518] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8368.061521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8368.061523] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8368.061531] Call Trace: [ 8368.061540] [ 8368.061544] ? show_regs.cold.9+0x22/0x2f [ 8368.061552] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8368.061558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8368.061562] ? watchdog+0x40/0x40 [ 8368.061566] ? hrtimer_interrupt+0x128/0x2c0 [ 8368.061570] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8368.061577] ? apic_timer_interrupt+0xf/0x20 [ 8368.061582] [ 8368.061585] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8368.061662] kthread+0x1d1/0x200 [ 8368.061668] ? set_kthread_struct+0x70/0x70 [ 8368.061673] ret_from_fork+0x1f/0x30 [ 8368.064909] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8368.064984] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8368.064991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8368.064994] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8368.065008] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8368.065012] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8368.065016] RAX: 0000000000006394 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8368.065019] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8368.065021] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8368.065024] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8368.065027] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8368.065031] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8368.065034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8368.065037] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8368.065045] Call Trace: [ 8368.068941] do_raw_spin_lock+0x177/0x190 [ 8368.068964] _raw_spin_lock+0x21/0x30 [ 8368.068976] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8368.069263] ? do_raw_spin_unlock+0x75/0x190 [ 8368.069273] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8368.069514] ? do_wait_intr+0xf0/0xf0 [ 8368.069523] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8368.069766] mdc_close+0x53b/0xd70 [mdc] [ 8368.069823] lmv_close+0x18d/0x490 [lmv] [ 8368.069866] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8368.070182] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8368.070264] ll_md_close+0x274/0x800 [lustre] [ 8368.070369] ll_file_release+0x128/0xc50 [lustre] [ 8368.070456] __fput+0x10d/0x440 [ 8368.070466] ____fput+0x12/0x20 [ 8368.070471] task_work_run+0xc6/0x110 [ 8368.070479] exit_to_usermode_loop+0x1dd/0x1f0 [ 8368.070487] do_syscall_64+0x3d6/0x440 [ 8368.070491] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8368.070498] RIP: 0033:0x7f1a63983cd8 [ 8368.070507] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8368.070512] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8368.070517] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8368.070519] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8368.070521] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8368.070524] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8368.070526] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8369.112948] [ 8369.114674] ? show_regs.cold.9+0x22/0x2f [ 8369.121297] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8369.132923] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8369.134514] ? watchdog+0x40/0x40 [ 8369.138881] ? hrtimer_interrupt+0x128/0x2c0 [ 8369.144954] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8369.149618] ? apic_timer_interrupt+0xf/0x20 [ 8369.152482] [ 8369.154357] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8369.159156] do_raw_spin_lock+0x177/0x190 [ 8369.161733] _raw_spin_lock+0x21/0x30 [ 8369.168679] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8369.170453] ? do_raw_spin_unlock+0x75/0x190 [ 8369.171507] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8369.191493] ? do_wait_intr+0xf0/0xf0 [ 8369.196141] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8369.200394] mdc_close+0x53b/0xd70 [mdc] [ 8369.206830] lmv_close+0x18d/0x490 [lmv] [ 8369.211013] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8369.214844] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8369.217681] ll_md_close+0x274/0x800 [lustre] [ 8369.222070] ll_file_release+0x128/0xc50 [lustre] [ 8369.225057] __fput+0x10d/0x440 [ 8369.227682] ____fput+0x12/0x20 [ 8369.229433] task_work_run+0xc6/0x110 [ 8369.233407] exit_to_usermode_loop+0x1dd/0x1f0 [ 8369.236915] do_syscall_64+0x3d6/0x440 [ 8369.240057] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8369.243472] RIP: 0033:0x7f1a63983cd8 [ 8369.247085] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8369.259407] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8369.264478] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8369.269321] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8369.278690] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8369.283868] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8369.293076] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8372.054588] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 8372.059599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8372.136063] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8372.146280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8372.155377] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8372.165777] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8372.184405] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8372.191289] RAX: 0000000000007c66 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8372.194841] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8372.199051] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8372.209155] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8372.223926] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8372.239485] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8372.244173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8372.248471] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8372.251155] Call Trace: [ 8372.259912] [ 8372.262393] ? show_regs.cold.9+0x22/0x2f [ 8372.265544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8372.269922] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8372.274611] ? watchdog+0x40/0x40 [ 8372.277290] ? hrtimer_interrupt+0x128/0x2c0 [ 8372.279622] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8372.283230] ? apic_timer_interrupt+0xf/0x20 [ 8372.285686] [ 8372.287151] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8372.292367] do_raw_spin_lock+0x177/0x190 [ 8372.294629] _raw_spin_lock+0x21/0x30 [ 8372.303701] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8372.308132] ? do_raw_spin_unlock+0x75/0x190 [ 8372.311279] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8372.317279] ? do_wait_intr+0xf0/0xf0 [ 8372.325251] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8372.333313] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8372.341452] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8372.353483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8372.357373] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8372.367173] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8372.377574] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8372.382440] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8372.385569] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8372.403036] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8372.408698] lmv_intent_open+0x338/0x1150 [lmv] [ 8372.412394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8372.422997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8372.426137] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8372.431812] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8372.434035] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8372.437434] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8372.443285] ? do_raw_spin_unlock+0x75/0x190 [ 8372.447360] ? _raw_spin_unlock+0x12/0x30 [ 8372.454360] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8372.456265] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8372.466019] ll_atomic_open+0x36a/0xd70 [lustre] [ 8372.470328] lookup_open+0x356/0xb40 [ 8372.474657] do_last+0x313/0xfc0 [ 8372.476570] ? nd_jump_root+0xe5/0x160 [ 8372.479264] ? path_init+0x437/0x520 [ 8372.485097] path_openat+0xf7/0x500 [ 8372.491377] ? do_raw_spin_unlock+0x75/0x190 [ 8372.502499] do_filp_open+0x99/0x140 [ 8372.505591] ? getname_flags+0x6e/0x330 [ 8372.509195] ? __check_object_size+0xff/0x256 [ 8372.512309] ? do_raw_spin_unlock+0x75/0x190 [ 8372.515961] ? _raw_spin_unlock+0x12/0x30 [ 8372.524431] do_sys_openat2+0x2b4/0x410 [ 8372.529458] do_sys_open+0x73/0xa0 [ 8372.532664] __x64_sys_openat+0x24/0x30 [ 8372.542446] do_syscall_64+0xc1/0x440 [ 8372.551353] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8372.556472] RIP: 0033:0x7f1a63983332 [ 8372.560885] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8372.574682] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8372.584955] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8372.593465] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8372.608501] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8372.623768] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8372.630489] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8375.282263] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6352s! [ 8375.290354] Showing busy workqueues and worker pools: [ 8375.304504] workqueue events: flags=0x0 [ 8375.305955] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8375.314059] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8375.326320] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8375.335240] pending: sec_gc_main [ptlrpc] [ 8375.343218] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8375.357204] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8375.365610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8375.383342] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8375.397829] workqueue events_long: flags=0x0 [ 8375.401830] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8375.406810] pending: lu_objects_flush [obdclass] [ 8375.411051] workqueue events_unbound: flags=0x2 [ 8375.415829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8375.425645] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8375.438606] workqueue events_power_efficient: flags=0x80 [ 8375.442386] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8375.447642] pending: neigh_periodic_work, do_cache_clean [ 8375.452858] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8375.456188] pending: neigh_periodic_work, check_lifetime [ 8375.458412] workqueue mm_percpu_wq: flags=0x8 [ 8375.460297] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8375.463566] pending: vmstat_update [ 8375.465544] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8375.470473] pending: vmstat_update [ 8375.472321] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8375.474669] pending: vmstat_update [ 8375.476282] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8375.481278] pending: vmstat_update [ 8375.483026] workqueue writeback: flags=0x4a [ 8375.484975] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8375.487638] pending: wb_workfn, wb_workfn [ 8375.489608] workqueue xprtiod: flags=0xa [ 8375.491603] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8375.497320] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8375.501549] workqueue ipv6_addrconf: flags=0x40008 [ 8375.504638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8375.512060] pending: addrconf_verify_work [ 8375.514938] workqueue knbd0-recv: flags=0x1a [ 8375.519574] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8375.524367] in-flight: 647:recv_work [ 8375.526258] workqueue ptlrpc_pinger: flags=0x2 [ 8375.529498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8375.535292] pending: ptlrpc_pinger_main [ptlrpc] [ 8375.539457] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8365s workers=2 idle: 652 [ 8382.576242] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 8382.596167] rcu: 3-...!: (4004633 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 8382.597906] (detected by 1, t=6360170 jiffies, g=269633, q=2272) [ 8382.599084] Sending NMI from CPU 1 to CPUs 3: [ 8382.602683] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 8382.603682] rcu: rcu_sched kthread starved for 6359816 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 8382.608721] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8382.613249] rcu: RCU grace-period kthread stack dump: [ 8382.617165] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8382.627237] Call Trace: [ 8382.630460] __schedule+0x351/0xcb0 [ 8382.632108] schedule+0xc0/0x180 [ 8382.634821] schedule_timeout+0xb4/0x190 [ 8382.637320] ? do_raw_spin_unlock+0x75/0x190 [ 8382.640845] ? __next_timer_interrupt+0x160/0x160 [ 8382.642800] ? prepare_to_swait_event+0xf3/0x140 [ 8382.644678] rcu_gp_kthread+0x8dd/0x15b0 [ 8382.646234] ? force_qs_rnp+0x2c0/0x2c0 [ 8382.650116] kthread+0x1d1/0x200 [ 8382.651441] ? set_kthread_struct+0x70/0x70 [ 8382.652686] ret_from_fork+0x1f/0x30 [ 8382.654127] rcu: Stack dump where RCU GP kthread last ran: [ 8382.656602] Sending NMI from CPU 1 to CPUs 0: [ 8382.662198] NMI backtrace for cpu 0 [ 8382.662206] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8382.662208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8382.662209] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8382.662211] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8382.662213] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 [ 8382.662215] RAX: 0000000000007466 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8382.662217] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8382.662218] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8382.662219] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8382.662221] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8382.662222] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8382.662223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8382.662225] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8382.662226] Call Trace: [ 8382.662227] [ 8382.662228] ? show_regs.cold.9+0x22/0x2f [ 8382.662229] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8382.662230] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8382.662231] ? nmi_handle+0xbd/0x210 [ 8382.662232] ? default_do_nmi+0x4b/0x1b0 [ 8382.662233] ? do_nmi+0x280/0x380 [ 8382.662234] ? end_repeat_nmi+0x16/0x1f [ 8382.662235] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8382.662236] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8382.662237] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8382.662238] [ 8382.662239] do_raw_spin_lock+0x177/0x190 [ 8382.662240] _raw_spin_lock+0x21/0x30 [ 8382.662241] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8382.662242] ? do_raw_spin_unlock+0x75/0x190 [ 8382.662243] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8382.662244] ? do_wait_intr+0xf0/0xf0 [ 8382.662246] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8382.662247] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8382.662248] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8382.662249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8382.662250] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8382.662251] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8382.662252] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8382.662253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8382.662254] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8382.662255] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8382.662257] lmv_intent_open+0x338/0x1150 [lmv] [ 8382.662258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8382.662259] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8382.662260] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8382.662261] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8382.662262] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8382.662264] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8382.662265] ? do_raw_spin_unlock+0x75/0x190 [ 8382.662266] ? _raw_spin_unlock+0x12/0x30 [ 8382.662267] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8382.662268] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8382.662269] ll_atomic_open+0x36a/0xd70 [lustre] [ 8382.662270] lookup_open+0x356/0xb40 [ 8382.662271] do_last+0x313/0xfc0 [ 8382.662272] ? nd_jump_root+0xe5/0x160 [ 8382.662273] ? path_init+0x437/0x520 [ 8382.662274] path_openat+0xf7/0x500 [ 8382.662275] ? do_raw_spin_unlock+0x75/0x190 [ 8382.662276] do_filp_open+0x99/0x140 [ 8382.662276] ? getname_flags+0x6e/0x330 [ 8382.662277] ? __check_object_size+0xff/0x256 [ 8382.662278] ? do_raw_spin_unlock+0x75/0x190 [ 8382.662279] ? _raw_spin_unlock+0x12/0x30 [ 8382.662281] do_sys_openat2+0x2b4/0x410 [ 8382.662281] do_sys_open+0x73/0xa0 [ 8382.662283] __x64_sys_openat+0x24/0x30 [ 8382.662284] do_syscall_64+0xc1/0x440 [ 8382.662285] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8382.662286] RIP: 0033:0x7f1a63983332 [ 8382.662287] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8382.662290] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8382.662292] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8382.662294] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8382.662296] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8382.662297] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8382.662298] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8396.057570] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8396.068162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8396.124349] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8396.150188] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8396.160213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8396.166532] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8396.186095] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8396.190765] RAX: 0000000000007d22 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8396.195331] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8396.204741] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8396.210227] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8396.214290] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8396.218289] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8396.225137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8396.230101] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8396.235393] Call Trace: [ 8396.238092] [ 8396.240103] ? show_regs.cold.9+0x22/0x2f [ 8396.242703] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8396.248162] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8396.250441] ? watchdog+0x40/0x40 [ 8396.258271] ? hrtimer_interrupt+0x128/0x2c0 [ 8396.263662] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8396.270928] ? apic_timer_interrupt+0xf/0x20 [ 8396.273244] [ 8396.278872] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8396.286506] do_raw_spin_lock+0x177/0x190 [ 8396.287998] _raw_spin_lock+0x21/0x30 [ 8396.293497] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8396.313622] ? do_raw_spin_unlock+0x75/0x190 [ 8396.320335] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8396.323329] ? do_wait_intr+0xf0/0xf0 [ 8396.330850] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8396.334357] mdc_close+0x53b/0xd70 [mdc] [ 8396.340644] lmv_close+0x18d/0x490 [lmv] [ 8396.342614] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8396.358366] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8396.363964] ll_md_close+0x274/0x800 [lustre] [ 8396.369816] ll_file_release+0x128/0xc50 [lustre] [ 8396.373715] __fput+0x10d/0x440 [ 8396.375371] ____fput+0x12/0x20 [ 8396.377266] task_work_run+0xc6/0x110 [ 8396.381169] exit_to_usermode_loop+0x1dd/0x1f0 [ 8396.384172] do_syscall_64+0x3d6/0x440 [ 8396.388705] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8396.394170] RIP: 0033:0x7f1a63983cd8 [ 8396.397622] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8396.412926] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8396.421310] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8396.429546] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8396.432397] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8396.435116] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8396.441442] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8405.987785] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6382s! [ 8405.993450] Showing busy workqueues and worker pools: [ 8405.995996] workqueue events: flags=0x0 [ 8405.998076] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8406.001260] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8406.004455] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8406.008645] pending: sec_gc_main [ptlrpc] [ 8406.012310] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8406.016850] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8406.021768] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8406.025112] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8406.030642] workqueue events_long: flags=0x0 [ 8406.032753] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8406.035579] pending: lu_objects_flush [obdclass] [ 8406.038613] workqueue events_unbound: flags=0x2 [ 8406.040868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8406.043832] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8406.048314] workqueue events_power_efficient: flags=0x80 [ 8406.051209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8406.054569] pending: neigh_periodic_work, do_cache_clean [ 8406.057488] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8406.060522] pending: neigh_periodic_work, check_lifetime [ 8406.064851] workqueue mm_percpu_wq: flags=0x8 [ 8406.067364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8406.070780] pending: vmstat_update [ 8406.072273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8406.075231] pending: vmstat_update [ 8406.076857] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8406.079887] pending: vmstat_update [ 8406.081618] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8406.084773] pending: vmstat_update [ 8406.086471] workqueue writeback: flags=0x4a [ 8406.088252] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8406.090668] pending: wb_workfn, wb_workfn [ 8406.092177] workqueue xprtiod: flags=0xa [ 8406.093783] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8406.096571] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8406.100068] workqueue ipv6_addrconf: flags=0x40008 [ 8406.102374] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8406.105145] pending: addrconf_verify_work [ 8406.107190] workqueue knbd0-recv: flags=0x1a [ 8406.109157] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8406.112246] in-flight: 647:recv_work [ 8406.114230] workqueue ptlrpc_pinger: flags=0x2 [ 8406.116208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8406.119004] pending: ptlrpc_pinger_main [ptlrpc] [ 8406.121181] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8396s workers=2 idle: 652 [ 8408.056080] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8408.059311] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8408.059644] Modules linked in: [ 8408.062945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8408.063016] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8408.063022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8408.063025] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8408.063036] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8408.063040] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8408.063043] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8408.063045] RDX: 0000000000004474 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8408.063047] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8408.063049] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8408.063052] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8408.063054] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8408.063056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8408.063057] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8408.063064] Call Trace: [ 8408.063068] [ 8408.063072] ? show_regs.cold.9+0x22/0x2f [ 8408.063079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8408.063083] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8408.063086] ? watchdog+0x40/0x40 [ 8408.063089] ? hrtimer_interrupt+0x128/0x2c0 [ 8408.063091] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8408.063097] ? apic_timer_interrupt+0xf/0x20 [ 8408.063101] [ 8408.063103] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8408.063106] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8408.063109] do_raw_spin_lock+0x177/0x190 [ 8408.063112] _raw_spin_lock+0x21/0x30 [ 8408.063115] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8408.063526] obd_mod_free+0x46/0x140 [ptlrpc] [ 8408.063718] mdc_commit_open+0x91/0xb0 [mdc] [ 8408.063753] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8408.063945] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8408.064139] after_reply+0xe2c/0x1780 [ptlrpc] [ 8408.064476] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8408.064669] ? __next_timer_interrupt+0x160/0x160 [ 8408.064679] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8408.064877] ? do_wait_intr+0xf0/0xf0 [ 8408.064886] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8408.065076] kthread+0x1d1/0x200 [ 8408.065082] ? set_kthread_struct+0x70/0x70 [ 8408.065085] ret_from_fork+0x1f/0x30 [ 8408.291457] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8408.328495] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8408.334619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8408.340459] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8408.347711] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8408.362818] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8408.368724] RAX: 0000000000000c0a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8408.373221] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8408.377884] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8408.380505] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8408.384032] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8408.386903] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8408.393433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8408.405934] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8408.417008] Call Trace: [ 8408.420493] [ 8408.425934] ? show_regs.cold.9+0x22/0x2f [ 8408.430952] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8408.435578] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8408.440872] ? watchdog+0x40/0x40 [ 8408.442877] ? hrtimer_interrupt+0x128/0x2c0 [ 8408.445851] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8408.448712] ? apic_timer_interrupt+0xf/0x20 [ 8408.451028] [ 8408.458355] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8408.467738] do_raw_spin_lock+0x177/0x190 [ 8408.470081] _raw_spin_lock+0x21/0x30 [ 8408.476730] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8408.487949] ? do_raw_spin_unlock+0x75/0x190 [ 8408.493108] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8408.499066] ? do_wait_intr+0xf0/0xf0 [ 8408.504128] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8408.511490] mdc_close+0x53b/0xd70 [mdc] [ 8408.518757] lmv_close+0x18d/0x490 [lmv] [ 8408.526723] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8408.530898] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8408.544791] ll_md_close+0x274/0x800 [lustre] [ 8408.558685] ll_file_release+0x128/0xc50 [lustre] [ 8408.569825] __fput+0x10d/0x440 [ 8408.574193] ____fput+0x12/0x20 [ 8408.575351] task_work_run+0xc6/0x110 [ 8408.582462] exit_to_usermode_loop+0x1dd/0x1f0 [ 8408.585340] do_syscall_64+0x3d6/0x440 [ 8408.594490] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8408.597623] RIP: 0033:0x7f1a63983cd8 [ 8408.600884] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8408.616741] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8408.619951] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8408.628936] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8408.633301] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8408.639028] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8408.645660] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8424.061006] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8424.069167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8424.121343] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8424.128853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8424.139408] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8424.142509] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8424.152739] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8424.156255] RAX: 0000000000004083 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8424.164940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8424.169565] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8424.177528] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8424.184965] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8424.193651] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8424.205138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8424.213920] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8424.221537] Call Trace: [ 8424.224922] [ 8424.225732] ? show_regs.cold.9+0x22/0x2f [ 8424.229950] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8424.232580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8424.240052] ? watchdog+0x40/0x40 [ 8424.243934] ? hrtimer_interrupt+0x128/0x2c0 [ 8424.248658] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8424.251148] ? apic_timer_interrupt+0xf/0x20 [ 8424.253281] [ 8424.254033] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8424.265176] do_raw_spin_lock+0x177/0x190 [ 8424.269747] _raw_spin_lock+0x21/0x30 [ 8424.274666] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8424.284881] ? do_raw_spin_unlock+0x75/0x190 [ 8424.287720] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8424.294991] ? do_wait_intr+0xf0/0xf0 [ 8424.299746] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8424.306245] mdc_close+0x53b/0xd70 [mdc] [ 8424.312163] lmv_close+0x18d/0x490 [lmv] [ 8424.315182] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8424.318969] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8424.325630] ll_md_close+0x274/0x800 [lustre] [ 8424.328661] ll_file_release+0x128/0xc50 [lustre] [ 8424.333650] __fput+0x10d/0x440 [ 8424.335626] ____fput+0x12/0x20 [ 8424.337590] task_work_run+0xc6/0x110 [ 8424.341252] exit_to_usermode_loop+0x1dd/0x1f0 [ 8424.344132] do_syscall_64+0x3d6/0x440 [ 8424.346245] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8424.349376] RIP: 0033:0x7f1a63983cd8 [ 8424.351612] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8424.376021] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8424.385312] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8424.393854] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8424.406236] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8424.414463] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8424.419937] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8432.054842] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8432.063416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8432.112716] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8432.132444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8432.144620] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8432.156101] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8432.172982] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8432.183108] RAX: 0000000000005c4a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8432.190608] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8432.201757] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8432.215847] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8432.222870] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8432.236662] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8432.245010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8432.253961] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8432.263233] Call Trace: [ 8432.266096] [ 8432.268066] ? show_regs.cold.9+0x22/0x2f [ 8432.276674] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8432.287759] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8432.294950] ? watchdog+0x40/0x40 [ 8432.296204] ? hrtimer_interrupt+0x128/0x2c0 [ 8432.302958] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8432.307970] ? apic_timer_interrupt+0xf/0x20 [ 8432.311036] [ 8432.312741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8432.320801] do_raw_spin_lock+0x177/0x190 [ 8432.323806] _raw_spin_lock+0x21/0x30 [ 8432.330090] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8432.342233] ? do_raw_spin_unlock+0x75/0x190 [ 8432.347463] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8432.355036] ? do_wait_intr+0xf0/0xf0 [ 8432.359890] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8432.364538] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8432.371456] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8432.382114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8432.387534] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8432.397102] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8432.404888] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8432.409317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8432.416479] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8432.426902] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8432.430973] lmv_intent_open+0x338/0x1150 [lmv] [ 8432.434536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8432.437451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8432.440276] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8432.443027] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8432.447966] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8432.452665] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8432.457683] ? do_raw_spin_unlock+0x75/0x190 [ 8432.459417] ? _raw_spin_unlock+0x12/0x30 [ 8432.460905] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8432.466628] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8432.472042] ll_atomic_open+0x36a/0xd70 [lustre] [ 8432.475107] lookup_open+0x356/0xb40 [ 8432.479030] do_last+0x313/0xfc0 [ 8432.481649] ? nd_jump_root+0xe5/0x160 [ 8432.483983] ? path_init+0x437/0x520 [ 8432.488037] path_openat+0xf7/0x500 [ 8432.489372] ? do_raw_spin_unlock+0x75/0x190 [ 8432.492376] do_filp_open+0x99/0x140 [ 8432.493805] ? getname_flags+0x6e/0x330 [ 8432.497831] ? __check_object_size+0xff/0x256 [ 8432.504231] ? do_raw_spin_unlock+0x75/0x190 [ 8432.506671] ? _raw_spin_unlock+0x12/0x30 [ 8432.508652] do_sys_openat2+0x2b4/0x410 [ 8432.512515] do_sys_open+0x73/0xa0 [ 8432.514978] __x64_sys_openat+0x24/0x30 [ 8432.516667] do_syscall_64+0xc1/0x440 [ 8432.518028] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8432.526712] RIP: 0033:0x7f1a63983332 [ 8432.533613] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8432.543107] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8432.549602] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8432.553408] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8432.556107] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8432.561923] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8432.566227] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8436.056655] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8436.058021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8436.059538] Modules linked in: [ 8436.065384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8436.065466] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8436.065473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8436.065476] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8436.065489] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8436.065493] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8436.065497] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8436.065499] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8436.065502] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8436.065504] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8436.065507] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8436.065510] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8436.065512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8436.065514] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8436.065524] Call Trace: [ 8436.065529] [ 8436.065534] ? show_regs.cold.9+0x22/0x2f [ 8436.065543] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8436.065549] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8436.065553] ? watchdog+0x40/0x40 [ 8436.065558] ? hrtimer_interrupt+0x128/0x2c0 [ 8436.065562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8436.065568] ? apic_timer_interrupt+0xf/0x20 [ 8436.065572] [ 8436.065575] ? native_safe_halt+0x2e/0x50 [ 8436.065579] kvm_wait+0xcd/0xe0 [ 8436.065586] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8436.065593] do_raw_spin_lock+0x177/0x190 [ 8436.065597] _raw_spin_lock+0x21/0x30 [ 8436.065602] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8436.067873] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 8436.088388] obd_mod_free+0x46/0x140 [ptlrpc] [ 8436.088736] mdc_commit_open+0x91/0xb0 [mdc] [ 8436.088772] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8436.089046] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8436.089253] after_reply+0xe2c/0x1780 [ptlrpc] [ 8436.089455] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8436.089734] ? __next_timer_interrupt+0x160/0x160 [ 8436.089744] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8436.090031] ? do_wait_intr+0xf0/0xf0 [ 8436.090041] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8436.090250] kthread+0x1d1/0x200 [ 8436.090256] ? set_kthread_struct+0x70/0x70 [ 8436.090260] ret_from_fork+0x1f/0x30 [ 8436.401666] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8436.454331] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8436.469730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8436.480410] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8436.484164] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8436.495080] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8436.511282] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8436.521838] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8436.535785] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8436.553906] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8436.559691] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8436.565317] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8436.570002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8436.576731] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8436.582002] Call Trace: [ 8436.583369] [ 8436.584685] ? show_regs.cold.9+0x22/0x2f [ 8436.597067] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8436.604794] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8436.609218] ? watchdog+0x40/0x40 [ 8436.614539] ? hrtimer_interrupt+0x128/0x2c0 [ 8436.617286] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8436.623765] ? apic_timer_interrupt+0xf/0x20 [ 8436.634476] [ 8436.637953] ? native_safe_halt+0x2e/0x50 [ 8436.639651] kvm_wait+0xcd/0xe0 [ 8436.640814] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8436.657813] do_raw_spin_lock+0x177/0x190 [ 8436.663490] _raw_spin_lock+0x21/0x30 [ 8436.664913] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8436.666910] ? do_raw_spin_unlock+0x75/0x190 [ 8436.677740] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8436.684302] ? do_wait_intr+0xf0/0xf0 [ 8436.689739] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8436.694192] mdc_close+0x53b/0xd70 [mdc] [ 8436.700365] lmv_close+0x18d/0x490 [lmv] [ 8436.701731] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8436.710929] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8436.719788] ll_md_close+0x274/0x800 [lustre] [ 8436.723050] ll_file_release+0x128/0xc50 [lustre] [ 8436.728259] __fput+0x10d/0x440 [ 8436.736810] ____fput+0x12/0x20 [ 8436.739268] task_work_run+0xc6/0x110 [ 8436.750875] exit_to_usermode_loop+0x1dd/0x1f0 [ 8436.752623] do_syscall_64+0x3d6/0x440 [ 8436.754003] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8436.770065] RIP: 0033:0x7f1a63983cd8 [ 8436.773377] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8436.803530] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8436.808411] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8436.818246] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8436.824443] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8436.833838] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8436.845473] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8436.852159] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6413s! [ 8436.855223] Showing busy workqueues and worker pools: [ 8436.868567] workqueue events: flags=0x0 [ 8436.871576] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8436.879197] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8436.884783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8436.890554] pending: sec_gc_main [ptlrpc] [ 8436.900732] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8436.906708] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8436.920802] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8436.930354] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8436.961065] workqueue events_long: flags=0x0 [ 8436.964785] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8436.972835] pending: lu_objects_flush [obdclass] [ 8436.984465] workqueue events_unbound: flags=0x2 [ 8436.994393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8437.003633] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8437.014905] workqueue events_power_efficient: flags=0x80 [ 8437.021746] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8437.025144] pending: neigh_periodic_work, do_cache_clean [ 8437.029851] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8437.037269] pending: neigh_periodic_work, check_lifetime [ 8437.039440] workqueue mm_percpu_wq: flags=0x8 [ 8437.044387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8437.057425] pending: vmstat_update [ 8437.063277] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8437.072660] pending: vmstat_update [ 8437.081866] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8437.090872] pending: vmstat_update [ 8437.094306] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8437.098644] pending: vmstat_update [ 8437.106351] workqueue writeback: flags=0x4a [ 8437.117988] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8437.132571] pending: wb_workfn, wb_workfn [ 8437.134963] workqueue xprtiod: flags=0xa [ 8437.137773] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8437.156688] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8437.170423] workqueue ipv6_addrconf: flags=0x40008 [ 8437.179205] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8437.181676] pending: addrconf_verify_work [ 8437.193805] workqueue knbd0-recv: flags=0x1a [ 8437.195475] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8437.202533] in-flight: 647:recv_work [ 8437.204063] workqueue ptlrpc_pinger: flags=0x2 [ 8437.212314] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8437.216607] pending: ptlrpc_pinger_main [ptlrpc] [ 8437.227206] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8427s workers=2 idle: 652 [ 8452.063804] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8452.076412] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8452.158068] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8452.172276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8452.180592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8452.185614] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8452.216373] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8452.231664] RAX: 000000000000708f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8452.249638] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8452.262707] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8452.270397] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8452.282208] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8452.291200] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8452.303228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8452.313411] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8452.325558] Call Trace: [ 8452.330900] [ 8452.337717] ? show_regs.cold.9+0x22/0x2f [ 8452.347869] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8452.357992] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8452.368157] ? watchdog+0x40/0x40 [ 8452.375544] ? hrtimer_interrupt+0x128/0x2c0 [ 8452.380200] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8452.386770] ? apic_timer_interrupt+0xf/0x20 [ 8452.391340] [ 8452.392687] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8452.396136] do_raw_spin_lock+0x177/0x190 [ 8452.398619] _raw_spin_lock+0x21/0x30 [ 8452.400382] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8452.403677] ? do_raw_spin_unlock+0x75/0x190 [ 8452.405903] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8452.409712] ? do_wait_intr+0xf0/0xf0 [ 8452.412626] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8452.417458] mdc_close+0x53b/0xd70 [mdc] [ 8452.420306] lmv_close+0x18d/0x490 [lmv] [ 8452.423075] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8452.427029] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8452.430211] ll_md_close+0x274/0x800 [lustre] [ 8452.432204] ll_file_release+0x128/0xc50 [lustre] [ 8452.434615] __fput+0x10d/0x440 [ 8452.436049] ____fput+0x12/0x20 [ 8452.437295] task_work_run+0xc6/0x110 [ 8452.439236] exit_to_usermode_loop+0x1dd/0x1f0 [ 8452.441895] do_syscall_64+0x3d6/0x440 [ 8452.451767] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8452.453645] RIP: 0033:0x7f1a63983cd8 [ 8452.455523] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8452.465725] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8452.469997] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8452.473523] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8452.477191] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8452.480926] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8452.484762] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8464.053025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8464.056011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8464.057606] Modules linked in: lustre(O) osp(O) [ 8464.060301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8464.060383] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8464.060391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8464.060394] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8464.060409] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8464.060413] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8464.060418] RAX: 0000000000000b50 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8464.060420] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8464.060423] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8464.060425] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8464.060427] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8464.060431] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8464.060434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8464.060437] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8464.060446] Call Trace: [ 8464.060456] [ 8464.060462] ? show_regs.cold.9+0x22/0x2f [ 8464.060472] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8464.060478] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8464.060482] ? watchdog+0x40/0x40 [ 8464.060487] ? hrtimer_interrupt+0x128/0x2c0 [ 8464.060491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8464.060500] ? apic_timer_interrupt+0xf/0x20 [ 8464.060504] [ 8464.060507] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8464.060514] do_raw_spin_lock+0x177/0x190 [ 8464.060519] _raw_spin_lock+0x21/0x30 [ 8464.060524] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8464.060885] ? do_raw_spin_unlock+0x75/0x190 [ 8464.060895] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8464.061165] ? do_wait_intr+0xf0/0xf0 [ 8464.061174] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8464.061382] mdc_close+0x53b/0xd70 [mdc] [ 8464.061421] lmv_close+0x18d/0x490 [lmv] [ 8464.061455] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8464.061570] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8464.061753] ll_md_close+0x274/0x800 [lustre] [ 8464.061848] ll_file_release+0x128/0xc50 [lustre] [ 8464.061936] __fput+0x10d/0x440 [ 8464.061946] ____fput+0x12/0x20 [ 8464.061950] task_work_run+0xc6/0x110 [ 8464.061958] exit_to_usermode_loop+0x1dd/0x1f0 [ 8464.061965] do_syscall_64+0x3d6/0x440 [ 8464.061970] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8464.061977] RIP: 0033:0x7f1a63983cd8 [ 8464.061984] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8464.061989] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8464.061993] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8464.061995] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8464.061998] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8464.062000] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8464.063267] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8464.068954] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8464.069351] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8464.069421] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8464.069427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8464.069430] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8464.069443] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8464.069447] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8464.069451] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8464.069454] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8464.069456] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8464.069458] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8464.069461] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8464.069464] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8464.069468] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8464.069470] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8464.069479] Call Trace: [ 8464.069485] [ 8464.069490] ? show_regs.cold.9+0x22/0x2f [ 8464.069500] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8464.069506] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8464.069510] ? watchdog+0x40/0x40 [ 8464.069514] ? hrtimer_interrupt+0x128/0x2c0 [ 8464.069519] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8464.069525] ? apic_timer_interrupt+0xf/0x20 [ 8464.069530] [ 8464.069533] ? native_safe_halt+0x2e/0x50 [ 8464.069537] kvm_wait+0xcd/0xe0 [ 8464.069545] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8464.069555] do_raw_spin_lock+0x177/0x190 [ 8464.069559] _raw_spin_lock+0x21/0x30 [ 8464.069563] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8464.070409] ? do_raw_spin_unlock+0x75/0x190 [ 8464.070416] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8464.070629] ? do_wait_intr+0xf0/0xf0 [ 8464.070638] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8464.071009] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8464.071230] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8464.072365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8464.072475] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8464.072516] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8464.072544] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8464.072570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8464.072653] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8464.073105] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8464.073135] lmv_intent_open+0x338/0x1150 [lmv] [ 8464.073166] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8464.073315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8464.073400] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8464.073424] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8464.073518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8464.073602] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8464.073687] ? do_raw_spin_unlock+0x75/0x190 [ 8464.073695] ? _raw_spin_unlock+0x12/0x30 [ 8464.073701] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8464.073727] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8464.073747] ll_atomic_open+0x36a/0xd70 [lustre] [ 8464.074313] lookup_open+0x356/0xb40 [ 8464.074322] do_last+0x313/0xfc0 [ 8464.074327] ? nd_jump_root+0xe5/0x160 [ 8464.074332] ? path_init+0x437/0x520 [ 8464.074336] path_openat+0xf7/0x500 [ 8464.074341] ? do_raw_spin_unlock+0x75/0x190 [ 8464.074346] do_filp_open+0x99/0x140 [ 8464.074352] ? getname_flags+0x6e/0x330 [ 8464.074356] ? __check_object_size+0xff/0x256 [ 8464.074363] ? do_raw_spin_unlock+0x75/0x190 [ 8464.074367] ? _raw_spin_unlock+0x12/0x30 [ 8464.074371] do_sys_openat2+0x2b4/0x410 [ 8464.074376] do_sys_open+0x73/0xa0 [ 8464.074380] __x64_sys_openat+0x24/0x30 [ 8464.074383] do_syscall_64+0xc1/0x440 [ 8464.074390] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8464.074396] RIP: 0033:0x7f1a63983332 [ 8464.074403] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8464.074407] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8464.074411] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8464.074414] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8464.074416] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8464.074419] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8464.074421] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8464.475181] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8464.489001] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8464.495087] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8464.498592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8464.501099] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8464.507491] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8464.510604] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8464.512805] RDX: 0000000000006ddd RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8464.515174] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8464.517395] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8464.519651] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8464.521687] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8464.524189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8464.526065] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8464.528607] Call Trace: [ 8464.529455] [ 8464.530162] ? show_regs.cold.9+0x22/0x2f [ 8464.531538] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8464.533345] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8464.534979] ? watchdog+0x40/0x40 [ 8464.536135] ? hrtimer_interrupt+0x128/0x2c0 [ 8464.537483] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8464.538959] ? apic_timer_interrupt+0xf/0x20 [ 8464.540312] [ 8464.540928] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8464.544615] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8464.546235] do_raw_spin_lock+0x177/0x190 [ 8464.547308] _raw_spin_lock+0x21/0x30 [ 8464.548297] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8464.552026] obd_mod_free+0x46/0x140 [ptlrpc] [ 8464.555079] mdc_commit_open+0x91/0xb0 [mdc] [ 8464.557031] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8464.561580] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8464.563940] after_reply+0xe2c/0x1780 [ptlrpc] [ 8464.566219] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8464.568114] ? __next_timer_interrupt+0x160/0x160 [ 8464.570666] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8464.573648] ? do_wait_intr+0xf0/0xf0 [ 8464.580641] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8464.585983] kthread+0x1d1/0x200 [ 8464.587416] ? set_kthread_struct+0x70/0x70 [ 8464.590782] ret_from_fork+0x1f/0x30 [ 8467.425107] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6444s! [ 8467.428597] Showing busy workqueues and worker pools: [ 8467.430389] workqueue events: flags=0x0 [ 8467.446666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8467.454653] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8467.459855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8467.464387] pending: sec_gc_main [ptlrpc] [ 8467.466364] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8467.479133] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8467.496133] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8467.502317] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8467.513177] workqueue events_long: flags=0x0 [ 8467.516305] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8467.523207] pending: lu_objects_flush [obdclass] [ 8467.531191] workqueue events_unbound: flags=0x2 [ 8467.535620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8467.549900] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8467.560995] workqueue events_power_efficient: flags=0x80 [ 8467.570899] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8467.580108] pending: neigh_periodic_work, do_cache_clean [ 8467.586008] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8467.591145] pending: neigh_periodic_work, check_lifetime [ 8467.594314] workqueue mm_percpu_wq: flags=0x8 [ 8467.599010] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8467.602682] pending: vmstat_update [ 8467.611704] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8467.625047] pending: vmstat_update [ 8467.627888] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8467.635297] pending: vmstat_update [ 8467.636705] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8467.645062] pending: vmstat_update [ 8467.650585] workqueue writeback: flags=0x4a [ 8467.655739] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8467.659904] pending: wb_workfn, wb_workfn [ 8467.663346] workqueue xprtiod: flags=0xa [ 8467.665492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8467.678060] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8467.692303] workqueue ipv6_addrconf: flags=0x40008 [ 8467.695144] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8467.707736] pending: addrconf_verify_work [ 8467.715725] workqueue knbd0-recv: flags=0x1a [ 8467.717610] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8467.727949] in-flight: 647:recv_work [ 8467.729480] workqueue ptlrpc_pinger: flags=0x2 [ 8467.740038] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8467.743696] pending: ptlrpc_pinger_main [ptlrpc] [ 8467.752533] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8457s workers=2 idle: 652 [ 8480.058193] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8480.066442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8480.099035] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8480.106683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8480.112315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8480.116361] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 8480.128158] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8480.132226] RAX: 0000000000006863 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8480.135861] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8480.140064] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8480.143583] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8480.147110] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8480.151139] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8480.155621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8480.161273] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8480.168846] Call Trace: [ 8480.170639] [ 8480.172690] ? show_regs.cold.9+0x22/0x2f [ 8480.174900] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8480.177428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8480.180602] ? watchdog+0x40/0x40 [ 8480.182828] ? hrtimer_interrupt+0x128/0x2c0 [ 8480.186086] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8480.190584] ? apic_timer_interrupt+0xf/0x20 [ 8480.194036] [ 8480.195613] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8480.198572] do_raw_spin_lock+0x177/0x190 [ 8480.202154] _raw_spin_lock+0x21/0x30 [ 8480.204655] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8480.208862] ? do_raw_spin_unlock+0x75/0x190 [ 8480.212055] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8480.217950] ? do_wait_intr+0xf0/0xf0 [ 8480.222629] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8480.227887] mdc_close+0x53b/0xd70 [mdc] [ 8480.231338] lmv_close+0x18d/0x490 [lmv] [ 8480.235594] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8480.241083] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8480.245140] ll_md_close+0x274/0x800 [lustre] [ 8480.249027] ll_file_release+0x128/0xc50 [lustre] [ 8480.253830] __fput+0x10d/0x440 [ 8480.257007] ____fput+0x12/0x20 [ 8480.259520] task_work_run+0xc6/0x110 [ 8480.262137] exit_to_usermode_loop+0x1dd/0x1f0 [ 8480.268060] do_syscall_64+0x3d6/0x440 [ 8480.271879] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8480.276480] RIP: 0033:0x7f1a63983cd8 [ 8480.279157] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8480.296747] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8480.302548] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8480.308139] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8480.314848] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8480.320448] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8480.326048] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8492.055747] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8492.057399] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8492.058057] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8492.058064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8492.058134] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8492.058139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8492.058143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8492.058158] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8492.058162] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8492.058166] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8492.058169] RDX: 0000000000002c74 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8492.058171] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8492.058173] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8492.058175] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8492.058179] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8492.058181] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8492.058183] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8492.058195] Call Trace: [ 8492.058201] [ 8492.058207] ? show_regs.cold.9+0x22/0x2f [ 8492.058218] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8492.058224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8492.058229] ? watchdog+0x40/0x40 [ 8492.058239] ? hrtimer_interrupt+0x128/0x2c0 [ 8492.058244] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8492.058252] ? apic_timer_interrupt+0xf/0x20 [ 8492.058257] [ 8492.058260] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8492.058264] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8492.058269] do_raw_spin_lock+0x177/0x190 [ 8492.058274] _raw_spin_lock+0x21/0x30 [ 8492.058279] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8492.058615] obd_mod_free+0x46/0x140 [ptlrpc] [ 8492.058829] mdc_commit_open+0x91/0xb0 [mdc] [ 8492.058877] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8492.059158] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8492.059378] after_reply+0xe2c/0x1780 [ptlrpc] [ 8492.059670] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8492.060004] ? __next_timer_interrupt+0x160/0x160 [ 8492.060013] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8492.060245] ? do_wait_intr+0xf0/0xf0 [ 8492.060255] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8492.060622] kthread+0x1d1/0x200 [ 8492.060628] ? set_kthread_struct+0x70/0x70 [ 8492.060632] ret_from_fork+0x1f/0x30 [ 8492.069095] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 8492.078100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8492.078187] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8492.078196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8492.078199] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8492.078213] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8492.078217] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8492.078222] RAX: 0000000000006398 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8492.078229] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8492.078231] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8492.078233] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8492.078235] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8492.078238] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8492.078241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8492.078243] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8492.078251] Call Trace: [ 8492.078257] [ 8492.078262] ? show_regs.cold.9+0x22/0x2f [ 8492.078276] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8492.078283] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8492.078287] ? watchdog+0x40/0x40 [ 8492.078292] ? hrtimer_interrupt+0x128/0x2c0 [ 8492.078296] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8492.078305] ? apic_timer_interrupt+0xf/0x20 [ 8492.078309] [ 8492.078312] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8492.078319] do_raw_spin_lock+0x177/0x190 [ 8492.078323] _raw_spin_lock+0x21/0x30 [ 8492.078328] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8492.078648] ? do_raw_spin_unlock+0x75/0x190 [ 8492.078655] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8492.095456] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [ 8492.135478] ? do_wait_intr+0xf0/0xf0 [ 8492.135499] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8492.135782] mdc_close+0x53b/0xd70 [mdc] [ 8492.135823] lmv_close+0x18d/0x490 [lmv] [ 8492.135859] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8492.138863] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8492.138956] ll_md_close+0x274/0x800 [lustre] [ 8492.139045] ll_file_release+0x128/0xc50 [lustre] [ 8492.139133] __fput+0x10d/0x440 [ 8492.139143] ____fput+0x12/0x20 [ 8492.139148] task_work_run+0xc6/0x110 [ 8492.158207] dm_mod sha512_ssse3 sha512_generic [ 8492.158225] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8492.158232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8492.158235] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8492.158250] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8492.158254] RSP: 0018:ffffc38541c37500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8492.158259] RAX: 0000000000005093 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8492.158262] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8492.158264] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8492.158266] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8492.158268] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8492.158272] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8492.158275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8492.158277] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8492.158285] Call Trace: [ 8492.158291] [ 8492.158297] ? show_regs.cold.9+0x22/0x2f [ 8492.158308] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8492.158315] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8492.158318] ? watchdog+0x40/0x40 [ 8492.158322] ? hrtimer_interrupt+0x128/0x2c0 [ 8492.158326] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8492.158334] ? apic_timer_interrupt+0xf/0x20 [ 8492.158339] [ 8492.158341] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8492.158347] do_raw_spin_lock+0x177/0x190 [ 8492.158352] _raw_spin_lock+0x21/0x30 [ 8492.158357] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8492.160477] ? do_raw_spin_unlock+0x75/0x190 [ 8492.160486] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8492.161208] ? do_wait_intr+0xf0/0xf0 [ 8492.161219] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8492.164788] exit_to_usermode_loop+0x1dd/0x1f0 [ 8492.164807] do_syscall_64+0x3d6/0x440 [ 8492.164812] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8492.164821] RIP: 0033:0x7f1a63983cd8 [ 8492.164830] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8492.164836] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8492.164841] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8492.164844] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8492.164846] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8492.164848] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8492.164850] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8492.598025] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8492.600651] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8492.603669] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8492.606770] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8492.609301] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8492.611721] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8492.613989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8492.616517] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8492.619378] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8492.623113] lmv_intent_open+0x338/0x1150 [lmv] [ 8492.625073] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8492.627865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8492.630580] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8492.632587] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8492.634873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8492.637285] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8492.639669] ? do_raw_spin_unlock+0x75/0x190 [ 8492.641533] ? _raw_spin_unlock+0x12/0x30 [ 8492.644060] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8492.647302] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8492.650283] ll_atomic_open+0x36a/0xd70 [lustre] [ 8492.659276] lookup_open+0x356/0xb40 [ 8492.665230] do_last+0x313/0xfc0 [ 8492.670299] ? nd_jump_root+0xe5/0x160 [ 8492.673043] ? path_init+0x437/0x520 [ 8492.675077] path_openat+0xf7/0x500 [ 8492.680740] ? do_raw_spin_unlock+0x75/0x190 [ 8492.687290] do_filp_open+0x99/0x140 [ 8492.690821] ? getname_flags+0x6e/0x330 [ 8492.692262] ? __check_object_size+0xff/0x256 [ 8492.694029] ? do_raw_spin_unlock+0x75/0x190 [ 8492.701155] ? _raw_spin_unlock+0x12/0x30 [ 8492.702892] do_sys_openat2+0x2b4/0x410 [ 8492.705272] do_sys_open+0x73/0xa0 [ 8492.708393] __x64_sys_openat+0x24/0x30 [ 8492.709933] do_syscall_64+0xc1/0x440 [ 8492.712963] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8492.715475] RIP: 0033:0x7f1a63983332 [ 8492.718041] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8492.738078] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8492.741776] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8492.744995] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8492.749478] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8492.754961] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8492.762366] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8498.146054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6475s! [ 8498.152017] Showing busy workqueues and worker pools: [ 8498.157642] workqueue events: flags=0x0 [ 8498.161440] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8498.164072] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8498.170476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8498.185257] pending: sec_gc_main [ptlrpc] [ 8498.187479] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8498.198440] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8498.217245] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8498.222483] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8498.227405] workqueue events_long: flags=0x0 [ 8498.236340] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8498.245865] pending: lu_objects_flush [obdclass] [ 8498.260345] workqueue events_unbound: flags=0x2 [ 8498.262195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8498.274301] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8498.288938] workqueue events_power_efficient: flags=0x80 [ 8498.294094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8498.298486] pending: neigh_periodic_work, do_cache_clean [ 8498.302904] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8498.309184] pending: neigh_periodic_work, check_lifetime [ 8498.316994] workqueue mm_percpu_wq: flags=0x8 [ 8498.322649] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8498.328661] pending: vmstat_update [ 8498.331633] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8498.338169] pending: vmstat_update [ 8498.342198] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8498.353069] pending: vmstat_update [ 8498.357671] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8498.363002] pending: vmstat_update [ 8498.365914] workqueue writeback: flags=0x4a [ 8498.369916] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8498.373820] pending: wb_workfn, wb_workfn [ 8498.375183] workqueue xprtiod: flags=0xa [ 8498.378388] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8498.384064] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8498.388927] workqueue ipv6_addrconf: flags=0x40008 [ 8498.397025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8498.408341] pending: addrconf_verify_work [ 8498.409987] workqueue knbd0-recv: flags=0x1a [ 8498.411468] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8498.420805] in-flight: 647:recv_work [ 8498.423580] workqueue ptlrpc_pinger: flags=0x2 [ 8498.425259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8498.433000] pending: ptlrpc_pinger_main [ptlrpc] [ 8498.438403] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8488s workers=2 idle: 652 [ 8508.057079] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8508.060701] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8508.092277] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8508.096745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8508.100717] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8508.102375] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8508.109463] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8508.112126] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8508.115237] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8508.122637] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8508.124674] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8508.135494] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8508.138700] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8508.142011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8508.144167] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8508.152469] Call Trace: [ 8508.153500] [ 8508.154476] ? show_regs.cold.9+0x22/0x2f [ 8508.156210] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8508.158132] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8508.161908] ? watchdog+0x40/0x40 [ 8508.164836] ? hrtimer_interrupt+0x128/0x2c0 [ 8508.169969] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8508.172588] ? apic_timer_interrupt+0xf/0x20 [ 8508.174638] [ 8508.175648] ? native_safe_halt+0x2e/0x50 [ 8508.177369] kvm_wait+0xcd/0xe0 [ 8508.178891] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8508.181744] do_raw_spin_lock+0x177/0x190 [ 8508.184127] _raw_spin_lock+0x21/0x30 [ 8508.185053] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8508.187347] ? do_raw_spin_unlock+0x75/0x190 [ 8508.189253] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8508.191715] ? do_wait_intr+0xf0/0xf0 [ 8508.193258] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8508.194825] mdc_close+0x53b/0xd70 [mdc] [ 8508.196081] lmv_close+0x18d/0x490 [lmv] [ 8508.198006] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8508.200558] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8508.202707] ll_md_close+0x274/0x800 [lustre] [ 8508.204714] ll_file_release+0x128/0xc50 [lustre] [ 8508.207196] __fput+0x10d/0x440 [ 8508.208567] ____fput+0x12/0x20 [ 8508.209834] task_work_run+0xc6/0x110 [ 8508.211654] exit_to_usermode_loop+0x1dd/0x1f0 [ 8508.213696] do_syscall_64+0x3d6/0x440 [ 8508.214878] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8508.216220] RIP: 0033:0x7f1a63983cd8 [ 8508.217119] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8508.223208] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8508.226922] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8508.228962] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8508.230724] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8508.232746] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8508.234736] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8520.056015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8520.060519] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8520.061240] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8520.090437] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8520.090453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8520.090457] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8520.090472] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8520.090476] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8520.090482] RAX: 0000000000002828 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8520.090485] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8520.090487] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8520.090489] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8520.090491] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8520.090495] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8520.090497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8520.090500] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8520.090509] Call Trace: [ 8520.090514] [ 8520.090519] ? show_regs.cold.9+0x22/0x2f [ 8520.090527] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8520.090533] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8520.090537] ? watchdog+0x40/0x40 [ 8520.090541] ? hrtimer_interrupt+0x128/0x2c0 [ 8520.090544] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8520.090556] ? apic_timer_interrupt+0xf/0x20 [ 8520.090561] [ 8520.090564] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8520.090571] do_raw_spin_lock+0x177/0x190 [ 8520.090576] _raw_spin_lock+0x21/0x30 [ 8520.090580] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8520.090979] ? do_raw_spin_unlock+0x75/0x190 [ 8520.090985] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8520.091186] ? do_wait_intr+0xf0/0xf0 [ 8520.091194] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8520.091411] mdc_close+0x53b/0xd70 [mdc] [ 8520.091512] lmv_close+0x18d/0x490 [lmv] [ 8520.091545] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8520.091657] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8520.091897] ll_md_close+0x274/0x800 [lustre] [ 8520.091979] ll_file_release+0x128/0xc50 [lustre] [ 8520.092061] __fput+0x10d/0x440 [ 8520.092070] ____fput+0x12/0x20 [ 8520.092074] task_work_run+0xc6/0x110 [ 8520.092081] exit_to_usermode_loop+0x1dd/0x1f0 [ 8520.092090] do_syscall_64+0x3d6/0x440 [ 8520.092094] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8520.092101] RIP: 0033:0x7f1a63983cd8 [ 8520.092108] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8520.092112] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8520.092117] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8520.092119] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8520.092121] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8520.092123] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8520.092126] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8520.296609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8520.310398] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8520.314218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8520.316660] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8520.318280] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8520.322768] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8520.324745] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8520.326671] RDX: 0000000000004506 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8520.328689] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8520.330892] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8520.332895] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8520.334721] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8520.338597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8520.340176] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8520.342129] Call Trace: [ 8520.343095] [ 8520.344083] ? show_regs.cold.9+0x22/0x2f [ 8520.346506] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8520.348426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8520.351368] ? watchdog+0x40/0x40 [ 8520.353419] ? hrtimer_interrupt+0x128/0x2c0 [ 8520.355083] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8520.360562] ? apic_timer_interrupt+0xf/0x20 [ 8520.363102] [ 8520.364021] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8520.367173] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8520.371767] do_raw_spin_lock+0x177/0x190 [ 8520.373963] _raw_spin_lock+0x21/0x30 [ 8520.375233] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8520.379165] obd_mod_free+0x46/0x140 [ptlrpc] [ 8520.381077] mdc_commit_open+0x91/0xb0 [mdc] [ 8520.382709] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8520.384291] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8520.390037] after_reply+0xe2c/0x1780 [ptlrpc] [ 8520.392800] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8520.394558] ? __next_timer_interrupt+0x160/0x160 [ 8520.395814] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8520.397565] ? do_wait_intr+0xf0/0xf0 [ 8520.398628] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8520.400431] kthread+0x1d1/0x200 [ 8520.401370] ? set_kthread_struct+0x70/0x70 [ 8520.403211] ret_from_fork+0x1f/0x30 [ 8524.053763] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8524.068739] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8524.104146] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8524.108282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8524.111805] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8524.114220] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8524.121638] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8524.124812] RAX: 0000000000007e0a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8524.127753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8524.130448] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8524.133185] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8524.135535] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8524.137804] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8524.140965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8524.143349] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8524.146102] Call Trace: [ 8524.147169] [ 8524.148212] ? show_regs.cold.9+0x22/0x2f [ 8524.149821] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8524.151678] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8524.153527] ? watchdog+0x40/0x40 [ 8524.154771] ? hrtimer_interrupt+0x128/0x2c0 [ 8524.156778] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8524.158747] ? apic_timer_interrupt+0xf/0x20 [ 8524.160653] [ 8524.161608] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8524.163870] do_raw_spin_lock+0x177/0x190 [ 8524.165301] _raw_spin_lock+0x21/0x30 [ 8524.166986] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8524.169003] ? do_raw_spin_unlock+0x75/0x190 [ 8524.170704] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8524.172927] ? do_wait_intr+0xf0/0xf0 [ 8524.174021] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8524.175994] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8524.177783] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8524.179879] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8524.181574] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8524.183695] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8524.185186] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8524.187062] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8524.190902] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8524.194895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8524.196826] lmv_intent_open+0x338/0x1150 [lmv] [ 8524.198528] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8524.200464] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8524.202417] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8524.203979] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8524.205829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8524.207840] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8524.209490] ? do_raw_spin_unlock+0x75/0x190 [ 8524.210935] ? _raw_spin_unlock+0x12/0x30 [ 8524.212399] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8524.214023] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8524.215795] ll_atomic_open+0x36a/0xd70 [lustre] [ 8524.217496] lookup_open+0x356/0xb40 [ 8524.218738] do_last+0x313/0xfc0 [ 8524.219920] ? nd_jump_root+0xe5/0x160 [ 8524.221015] ? path_init+0x437/0x520 [ 8524.222317] path_openat+0xf7/0x500 [ 8524.223505] ? do_raw_spin_unlock+0x75/0x190 [ 8524.224962] do_filp_open+0x99/0x140 [ 8524.226328] ? getname_flags+0x6e/0x330 [ 8524.227698] ? __check_object_size+0xff/0x256 [ 8524.229175] ? do_raw_spin_unlock+0x75/0x190 [ 8524.230761] ? _raw_spin_unlock+0x12/0x30 [ 8524.232190] do_sys_openat2+0x2b4/0x410 [ 8524.233543] do_sys_open+0x73/0xa0 [ 8524.234629] __x64_sys_openat+0x24/0x30 [ 8524.235979] do_syscall_64+0xc1/0x440 [ 8524.237173] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8524.238887] RIP: 0033:0x7f1a63983332 [ 8524.239995] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8524.246549] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8524.248777] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8524.250823] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8524.253479] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8524.256065] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8524.258441] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8528.868432] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6505s! [ 8528.875476] Showing busy workqueues and worker pools: [ 8528.883148] workqueue events: flags=0x0 [ 8528.888646] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8528.897016] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8528.908118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8528.914930] pending: sec_gc_main [ptlrpc] [ 8528.917312] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8528.921017] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8528.925086] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8528.928464] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8528.939046] workqueue events_long: flags=0x0 [ 8528.944169] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8528.951748] pending: lu_objects_flush [obdclass] [ 8528.955811] workqueue events_unbound: flags=0x2 [ 8528.957769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8528.967749] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8528.975568] workqueue events_power_efficient: flags=0x80 [ 8528.981925] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8528.986901] pending: neigh_periodic_work, do_cache_clean [ 8528.990147] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8528.996549] pending: neigh_periodic_work, check_lifetime [ 8528.999359] workqueue mm_percpu_wq: flags=0x8 [ 8529.000949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8529.007062] pending: vmstat_update [ 8529.011891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8529.015905] pending: vmstat_update [ 8529.017428] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8529.025040] pending: vmstat_update [ 8529.031044] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8529.036001] pending: vmstat_update [ 8529.039788] workqueue writeback: flags=0x4a [ 8529.042991] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8529.045794] pending: wb_workfn, wb_workfn [ 8529.048576] workqueue xprtiod: flags=0xa [ 8529.052109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8529.058825] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8529.062515] workqueue ipv6_addrconf: flags=0x40008 [ 8529.065651] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8529.070001] pending: addrconf_verify_work [ 8529.073991] workqueue knbd0-recv: flags=0x1a [ 8529.085279] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8529.088009] in-flight: 647:recv_work [ 8529.092126] workqueue ptlrpc_pinger: flags=0x2 [ 8529.101911] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8529.110170] pending: ptlrpc_pinger_main [ptlrpc] [ 8529.118042] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8519s workers=2 idle: 652 [ 8536.061004] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8536.065216] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8536.079233] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8536.083187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8536.086356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8536.088514] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8536.094548] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8536.097355] RAX: 0000000000003284 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8536.099445] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8536.101529] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8536.104042] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8536.106013] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8536.108187] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8536.110475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8536.112720] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8536.114665] Call Trace: [ 8536.115521] [ 8536.116262] ? show_regs.cold.9+0x22/0x2f [ 8536.117429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8536.119081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8536.120386] ? watchdog+0x40/0x40 [ 8536.121484] ? hrtimer_interrupt+0x128/0x2c0 [ 8536.122869] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8536.124630] ? apic_timer_interrupt+0xf/0x20 [ 8536.126297] [ 8536.126923] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8536.128876] do_raw_spin_lock+0x177/0x190 [ 8536.130389] _raw_spin_lock+0x21/0x30 [ 8536.131969] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8536.134114] ? do_raw_spin_unlock+0x75/0x190 [ 8536.135550] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8536.137595] ? do_wait_intr+0xf0/0xf0 [ 8536.138963] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8536.140879] mdc_close+0x53b/0xd70 [mdc] [ 8536.142069] lmv_close+0x18d/0x490 [lmv] [ 8536.143830] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8536.145805] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8536.147407] ll_md_close+0x274/0x800 [lustre] [ 8536.149367] ll_file_release+0x128/0xc50 [lustre] [ 8536.150877] __fput+0x10d/0x440 [ 8536.152187] ____fput+0x12/0x20 [ 8536.153040] task_work_run+0xc6/0x110 [ 8536.154331] exit_to_usermode_loop+0x1dd/0x1f0 [ 8536.155830] do_syscall_64+0x3d6/0x440 [ 8536.157061] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8536.158924] RIP: 0033:0x7f1a63983cd8 [ 8536.160307] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8536.166853] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8536.169801] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8536.171970] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8536.174757] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8536.177289] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8536.179633] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8548.056033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8548.059947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8548.060251] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8548.082872] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8548.082943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8548.082946] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8548.082959] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8548.082963] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8548.082969] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8548.082971] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8548.082974] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8548.082976] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8548.082978] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8548.082981] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8548.082984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8548.082987] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8548.082994] Call Trace: [ 8548.082999] [ 8548.083061] ? show_regs.cold.9+0x22/0x2f [ 8548.083071] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8548.083078] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8548.083083] ? watchdog+0x40/0x40 [ 8548.083088] ? hrtimer_interrupt+0x128/0x2c0 [ 8548.083092] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8548.083098] ? apic_timer_interrupt+0xf/0x20 [ 8548.083103] [ 8548.083106] ? native_safe_halt+0x2e/0x50 [ 8548.083110] kvm_wait+0xcd/0xe0 [ 8548.083116] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8548.083125] do_raw_spin_lock+0x177/0x190 [ 8548.083130] _raw_spin_lock+0x21/0x30 [ 8548.083134] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8548.083568] ? do_raw_spin_unlock+0x75/0x190 [ 8548.083573] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8548.083800] ? do_wait_intr+0xf0/0xf0 [ 8548.083811] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8548.088603] Modules linked in: [ 8548.094072] mdc_close+0x53b/0xd70 [mdc] [ 8548.094126] lmv_close+0x18d/0x490 [lmv] [ 8548.094155] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8548.094330] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8548.094523] ll_md_close+0x274/0x800 [lustre] [ 8548.094612] ll_file_release+0x128/0xc50 [lustre] [ 8548.094703] __fput+0x10d/0x440 [ 8548.094713] ____fput+0x12/0x20 [ 8548.094718] task_work_run+0xc6/0x110 [ 8548.094725] exit_to_usermode_loop+0x1dd/0x1f0 [ 8548.094733] do_syscall_64+0x3d6/0x440 [ 8548.094737] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8548.094746] RIP: 0033:0x7f1a63983cd8 [ 8548.094754] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8548.094758] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8548.094763] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8548.094766] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8548.094768] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8548.094771] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8548.094773] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8548.293761] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8548.337622] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8548.343980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8548.349349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8548.352862] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8548.367733] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8548.373571] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8548.376354] RDX: 0000000000007e83 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8548.379721] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8548.383060] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8548.386657] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8548.390308] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8548.396233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8548.398105] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8548.405533] Call Trace: [ 8548.407399] [ 8548.411443] ? show_regs.cold.9+0x22/0x2f [ 8548.412927] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8548.416612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8548.418605] ? watchdog+0x40/0x40 [ 8548.420080] ? hrtimer_interrupt+0x128/0x2c0 [ 8548.424575] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8548.427331] ? apic_timer_interrupt+0xf/0x20 [ 8548.429224] [ 8548.430005] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8548.436228] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8548.439323] do_raw_spin_lock+0x177/0x190 [ 8548.441917] _raw_spin_lock+0x21/0x30 [ 8548.446310] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8548.448681] obd_mod_free+0x46/0x140 [ptlrpc] [ 8548.452027] mdc_commit_open+0x91/0xb0 [mdc] [ 8548.454691] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8548.457560] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8548.460771] after_reply+0xe2c/0x1780 [ptlrpc] [ 8548.462150] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8548.465303] ? __next_timer_interrupt+0x160/0x160 [ 8548.466857] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8548.472178] ? do_wait_intr+0xf0/0xf0 [ 8548.475696] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8548.480655] kthread+0x1d1/0x200 [ 8548.482451] ? set_kthread_struct+0x70/0x70 [ 8548.488657] ret_from_fork+0x1f/0x30 [ 8556.053095] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8556.056247] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8556.089985] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8556.103024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8556.108608] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8556.113434] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8556.137505] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8556.143212] RAX: 000000000000608f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8556.149775] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8556.159630] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8556.165327] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8556.179853] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8556.186563] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8556.195386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8556.208544] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8556.216138] Call Trace: [ 8556.217130] [ 8556.217600] ? show_regs.cold.9+0x22/0x2f [ 8556.219560] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8556.222198] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8556.228375] ? watchdog+0x40/0x40 [ 8556.230530] ? hrtimer_interrupt+0x128/0x2c0 [ 8556.233202] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8556.242122] ? apic_timer_interrupt+0xf/0x20 [ 8556.245535] [ 8556.247036] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8556.249552] do_raw_spin_lock+0x177/0x190 [ 8556.262296] _raw_spin_lock+0x21/0x30 [ 8556.264404] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8556.268444] ? do_raw_spin_unlock+0x75/0x190 [ 8556.273223] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8556.282402] ? do_wait_intr+0xf0/0xf0 [ 8556.288407] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8556.296211] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8556.299434] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8556.308488] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8556.316801] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8556.323275] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8556.327811] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8556.332499] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8556.337359] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8556.341417] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8556.344370] lmv_intent_open+0x338/0x1150 [lmv] [ 8556.352118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8556.355554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8556.358179] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8556.360495] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8556.366058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8556.374485] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8556.384022] ? do_raw_spin_unlock+0x75/0x190 [ 8556.385567] ? _raw_spin_unlock+0x12/0x30 [ 8556.397424] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8556.414681] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8556.421591] ll_atomic_open+0x36a/0xd70 [lustre] [ 8556.431832] lookup_open+0x356/0xb40 [ 8556.434356] do_last+0x313/0xfc0 [ 8556.444176] ? nd_jump_root+0xe5/0x160 [ 8556.445783] ? path_init+0x437/0x520 [ 8556.447632] path_openat+0xf7/0x500 [ 8556.449888] ? do_raw_spin_unlock+0x75/0x190 [ 8556.452131] do_filp_open+0x99/0x140 [ 8556.454330] ? getname_flags+0x6e/0x330 [ 8556.456805] ? __check_object_size+0xff/0x256 [ 8556.460151] ? do_raw_spin_unlock+0x75/0x190 [ 8556.462593] ? _raw_spin_unlock+0x12/0x30 [ 8556.466563] do_sys_openat2+0x2b4/0x410 [ 8556.469155] do_sys_open+0x73/0xa0 [ 8556.471737] __x64_sys_openat+0x24/0x30 [ 8556.474240] do_syscall_64+0xc1/0x440 [ 8556.476214] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8556.479180] RIP: 0033:0x7f1a63983332 [ 8556.485339] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8556.500172] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8556.508422] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8556.522972] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8556.527339] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8556.535898] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8556.551238] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8559.586256] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6536s! [ 8559.589173] Showing busy workqueues and worker pools: [ 8559.601141] workqueue events: flags=0x0 [ 8559.604147] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8559.611757] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8559.621395] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8559.628548] pending: sec_gc_main [ptlrpc] [ 8559.635656] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8559.642701] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8559.649719] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8559.669105] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8559.684395] workqueue events_long: flags=0x0 [ 8559.689170] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8559.694110] pending: lu_objects_flush [obdclass] [ 8559.703588] workqueue events_unbound: flags=0x2 [ 8559.712551] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8559.717993] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8559.737556] workqueue events_power_efficient: flags=0x80 [ 8559.741584] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8559.748402] pending: neigh_periodic_work, do_cache_clean [ 8559.756807] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8559.772770] pending: neigh_periodic_work, check_lifetime [ 8559.780564] workqueue mm_percpu_wq: flags=0x8 [ 8559.788671] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8559.795938] pending: vmstat_update [ 8559.802111] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8559.807657] pending: vmstat_update [ 8559.809674] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8559.822425] pending: vmstat_update [ 8559.834482] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8559.841685] pending: vmstat_update [ 8559.852602] workqueue writeback: flags=0x4a [ 8559.855718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8559.860470] pending: wb_workfn, wb_workfn [ 8559.861760] workqueue xprtiod: flags=0xa [ 8559.868261] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8559.872656] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8559.883324] workqueue ipv6_addrconf: flags=0x40008 [ 8559.890292] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8559.893490] pending: addrconf_verify_work [ 8559.895952] workqueue knbd0-recv: flags=0x1a [ 8559.898187] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8559.902499] in-flight: 647:recv_work [ 8559.904588] workqueue ptlrpc_pinger: flags=0x2 [ 8559.908779] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8559.912485] pending: ptlrpc_pinger_main [ptlrpc] [ 8559.921261] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8549s workers=2 idle: 652 [ 8562.581029] rcu: INFO: rcu_sched self-detected stall on CPU [ 8562.583559] rcu: 3-...!: (4111186 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 8562.592410] (t=6540185 jiffies g=269633 q=2360) [ 8562.595110] rcu: rcu_sched kthread starved for 6539833 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 8562.605593] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8562.616457] rcu: RCU grace-period kthread stack dump: [ 8562.620157] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8562.626730] Call Trace: [ 8562.628121] __schedule+0x351/0xcb0 [ 8562.630273] schedule+0xc0/0x180 [ 8562.631749] schedule_timeout+0xb4/0x190 [ 8562.635541] ? do_raw_spin_unlock+0x75/0x190 [ 8562.642466] ? __next_timer_interrupt+0x160/0x160 [ 8562.651197] ? prepare_to_swait_event+0xf3/0x140 [ 8562.656288] rcu_gp_kthread+0x8dd/0x15b0 [ 8562.659538] ? force_qs_rnp+0x2c0/0x2c0 [ 8562.660974] kthread+0x1d1/0x200 [ 8562.668270] ? set_kthread_struct+0x70/0x70 [ 8562.674394] ret_from_fork+0x1f/0x30 [ 8562.677628] rcu: Stack dump where RCU GP kthread last ran: [ 8562.684578] Sending NMI from CPU 3 to CPUs 1: [ 8562.696026] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 7.832 msecs [ 8562.697939] NMI backtrace for cpu 1 [ 8562.697947] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8562.697949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8562.697951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8562.697953] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8562.697955] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 [ 8562.697957] RAX: 0000000000004c9b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8562.697959] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8562.697960] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8562.697962] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8562.697963] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8562.697964] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8562.697965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8562.697967] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8562.698177] Call Trace: [ 8562.698179] [ 8562.698180] ? show_regs.cold.9+0x22/0x2f [ 8562.698181] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8562.698182] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8562.698183] ? nmi_handle+0xbd/0x210 [ 8562.698184] ? default_do_nmi+0x4b/0x1b0 [ 8562.698185] ? do_nmi+0x280/0x380 [ 8562.698186] ? end_repeat_nmi+0x16/0x1f [ 8562.698187] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8562.698189] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8562.698190] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8562.698191] [ 8562.698192] do_raw_spin_lock+0x177/0x190 [ 8562.698193] _raw_spin_lock+0x21/0x30 [ 8562.698194] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8562.698195] ? do_raw_spin_unlock+0x75/0x190 [ 8562.698196] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8562.698197] ? do_wait_intr+0xf0/0xf0 [ 8562.698198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8562.698199] mdc_close+0x53b/0xd70 [mdc] [ 8562.698200] lmv_close+0x18d/0x490 [lmv] [ 8562.698201] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8562.698203] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8562.698204] ll_md_close+0x274/0x800 [lustre] [ 8562.698205] ll_file_release+0x128/0xc50 [lustre] [ 8562.698206] __fput+0x10d/0x440 [ 8562.698207] ____fput+0x12/0x20 [ 8562.698208] task_work_run+0xc6/0x110 [ 8562.698209] exit_to_usermode_loop+0x1dd/0x1f0 [ 8562.698210] do_syscall_64+0x3d6/0x440 [ 8562.698211] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8562.698212] RIP: 0033:0x7f1a63983cd8 [ 8562.698214] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8562.698216] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8562.698218] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8562.698219] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8562.698221] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8562.698222] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8562.698224] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8562.698229] NMI backtrace for cpu 3 [ 8563.089461] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8563.099727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8563.103469] Call Trace: [ 8563.104446] [ 8563.105530] ? dump_stack+0xbb/0x10e [ 8563.107560] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8563.109930] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8563.112275] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8563.114498] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8563.117634] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8563.123935] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8563.126274] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8563.129251] ? update_process_times+0x6d/0xc0 [ 8563.131590] ? tick_sched_handle.isra.18+0x27/0x90 [ 8563.135112] ? tick_sched_timer+0x53/0xb0 [ 8563.137406] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8563.140125] ? tick_sched_handle.isra.18+0x90/0x90 [ 8563.142172] ? hrtimer_interrupt+0x128/0x2c0 [ 8563.144062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8563.146518] ? apic_timer_interrupt+0xf/0x20 [ 8563.149194] [ 8563.150257] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8563.152885] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8563.155611] ? do_raw_spin_lock+0x177/0x190 [ 8563.158257] ? _raw_spin_lock+0x21/0x30 [ 8563.160268] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8563.163324] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8563.166331] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8563.168514] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8563.172407] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8563.175848] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8563.178776] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8563.181577] ? __next_timer_interrupt+0x160/0x160 [ 8563.184164] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8563.186535] ? do_wait_intr+0xf0/0xf0 [ 8563.188595] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8563.192630] ? kthread+0x1d1/0x200 [ 8563.194432] ? set_kthread_struct+0x70/0x70 [ 8563.196901] ? ret_from_fork+0x1f/0x30 [ 8564.064023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8564.090141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8564.200198] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8564.216217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8564.230172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8564.243324] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8564.268387] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8564.284363] RAX: 0000000000006c93 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8564.296458] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8564.307387] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8564.320314] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8564.327398] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8564.335621] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8564.351867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8564.353664] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8564.370295] Call Trace: [ 8564.379552] [ 8564.380400] ? show_regs.cold.9+0x22/0x2f [ 8564.384955] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8564.394038] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8564.400274] ? watchdog+0x40/0x40 [ 8564.403584] ? hrtimer_interrupt+0x128/0x2c0 [ 8564.407569] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8564.422335] ? apic_timer_interrupt+0xf/0x20 [ 8564.430085] [ 8564.431894] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8564.442342] do_raw_spin_lock+0x177/0x190 [ 8564.451107] _raw_spin_lock+0x21/0x30 [ 8564.459008] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8564.470093] ? do_raw_spin_unlock+0x75/0x190 [ 8564.477042] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8564.485151] ? do_wait_intr+0xf0/0xf0 [ 8564.492739] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8564.496922] mdc_close+0x53b/0xd70 [mdc] [ 8564.505610] lmv_close+0x18d/0x490 [lmv] [ 8564.512814] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8564.524165] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8564.533346] ll_md_close+0x274/0x800 [lustre] [ 8564.538966] ll_file_release+0x128/0xc50 [lustre] [ 8564.546300] __fput+0x10d/0x440 [ 8564.547323] ____fput+0x12/0x20 [ 8564.548429] task_work_run+0xc6/0x110 [ 8564.554114] exit_to_usermode_loop+0x1dd/0x1f0 [ 8564.563393] do_syscall_64+0x3d6/0x440 [ 8564.568219] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8564.572400] RIP: 0033:0x7f1a63983cd8 [ 8564.575975] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8564.600169] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8564.616321] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8564.624535] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8564.627933] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8564.633673] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8564.643212] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8584.054502] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8584.065240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8584.107609] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8584.120321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8584.131931] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8584.134104] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8584.160091] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8584.169429] RAX: 0000000000007d0c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8584.172365] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8584.183046] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8584.187748] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8584.196249] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8584.205035] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8584.209318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8584.212597] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8584.220451] Call Trace: [ 8584.222008] [ 8584.222823] ? show_regs.cold.9+0x22/0x2f [ 8584.229438] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8584.232140] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8584.237412] ? watchdog+0x40/0x40 [ 8584.239224] ? hrtimer_interrupt+0x128/0x2c0 [ 8584.240764] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8584.247067] ? apic_timer_interrupt+0xf/0x20 [ 8584.250112] [ 8584.251949] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8584.254732] do_raw_spin_lock+0x177/0x190 [ 8584.256482] _raw_spin_lock+0x21/0x30 [ 8584.263498] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8584.268376] ? do_raw_spin_unlock+0x75/0x190 [ 8584.272581] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8584.278244] ? do_wait_intr+0xf0/0xf0 [ 8584.280151] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8584.284235] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8584.287180] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8584.290298] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8584.292362] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8584.296214] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8584.298439] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8584.300404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8584.303158] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8584.306197] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8584.309806] lmv_intent_open+0x338/0x1150 [lmv] [ 8584.312026] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8584.314968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8584.317619] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8584.319922] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8584.322442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8584.325427] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8584.328180] ? do_raw_spin_unlock+0x75/0x190 [ 8584.330305] ? _raw_spin_unlock+0x12/0x30 [ 8584.332477] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8584.335340] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8584.338465] ll_atomic_open+0x36a/0xd70 [lustre] [ 8584.342505] lookup_open+0x356/0xb40 [ 8584.344624] do_last+0x313/0xfc0 [ 8584.347191] ? nd_jump_root+0xe5/0x160 [ 8584.361826] ? path_init+0x437/0x520 [ 8584.363335] path_openat+0xf7/0x500 [ 8584.364494] ? do_raw_spin_unlock+0x75/0x190 [ 8584.372525] do_filp_open+0x99/0x140 [ 8584.377403] ? getname_flags+0x6e/0x330 [ 8584.382148] ? __check_object_size+0xff/0x256 [ 8584.384865] ? do_raw_spin_unlock+0x75/0x190 [ 8584.387504] ? _raw_spin_unlock+0x12/0x30 [ 8584.389994] do_sys_openat2+0x2b4/0x410 [ 8584.392055] do_sys_open+0x73/0xa0 [ 8584.395046] __x64_sys_openat+0x24/0x30 [ 8584.398050] do_syscall_64+0xc1/0x440 [ 8584.401466] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8584.404031] RIP: 0033:0x7f1a63983332 [ 8584.406206] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8584.430990] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8584.433609] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8584.437523] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8584.444098] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8584.447802] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8584.451722] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8588.057277] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 8588.058676] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8588.062374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8588.062457] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8588.062465] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8588.062468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8588.062482] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8588.062486] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8588.062491] RAX: 0000000000004b8f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8588.062493] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8588.062496] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8588.062498] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8588.062500] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8588.062504] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8588.062507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8588.062510] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8588.062519] Call Trace: [ 8588.062526] [ 8588.062532] ? show_regs.cold.9+0x22/0x2f [ 8588.062542] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8588.062550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8588.062554] ? watchdog+0x40/0x40 [ 8588.062558] ? hrtimer_interrupt+0x128/0x2c0 [ 8588.062562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8588.062572] ? apic_timer_interrupt+0xf/0x20 [ 8588.062577] [ 8588.062580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8588.062586] do_raw_spin_lock+0x177/0x190 [ 8588.062591] _raw_spin_lock+0x21/0x30 [ 8588.062596] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8588.063349] ? do_raw_spin_unlock+0x75/0x190 [ 8588.063357] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8588.063555] ? do_wait_intr+0xf0/0xf0 [ 8588.063563] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8588.063774] mdc_close+0x53b/0xd70 [mdc] [ 8588.063813] lmv_close+0x18d/0x490 [lmv] [ 8588.063857] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8588.070952] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [ 8588.104937] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8588.105079] ll_md_close+0x274/0x800 [lustre] [ 8588.105162] ll_file_release+0x128/0xc50 [lustre] [ 8588.105243] __fput+0x10d/0x440 [ 8588.105256] ____fput+0x12/0x20 [ 8588.105263] task_work_run+0xc6/0x110 [ 8588.105271] exit_to_usermode_loop+0x1dd/0x1f0 [ 8588.105279] do_syscall_64+0x3d6/0x440 [ 8588.105283] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8588.105292] RIP: 0033:0x7f1a63983cd8 [ 8588.105300] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8588.105304] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8588.105310] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8588.105313] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8588.105315] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8588.105317] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8588.105319] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8588.255845] sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8588.263305] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8588.266982] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8588.270862] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8588.272932] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8588.280514] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8588.283816] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8588.287072] RDX: 0000000000007c39 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8588.290945] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8588.294239] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8588.299388] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8588.304227] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8588.308167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8588.312519] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8588.323309] Call Trace: [ 8588.325005] [ 8588.326341] ? show_regs.cold.9+0x22/0x2f [ 8588.329252] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8588.331415] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8588.335309] ? watchdog+0x40/0x40 [ 8588.339897] ? hrtimer_interrupt+0x128/0x2c0 [ 8588.342146] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8588.348781] ? apic_timer_interrupt+0xf/0x20 [ 8588.355786] [ 8588.358190] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8588.366605] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8588.381610] do_raw_spin_lock+0x177/0x190 [ 8588.388474] _raw_spin_lock+0x21/0x30 [ 8588.392377] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8588.401575] obd_mod_free+0x46/0x140 [ptlrpc] [ 8588.408148] mdc_commit_open+0x91/0xb0 [mdc] [ 8588.416337] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8588.422927] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8588.432869] after_reply+0xe2c/0x1780 [ptlrpc] [ 8588.439205] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8588.447168] ? __next_timer_interrupt+0x160/0x160 [ 8588.449063] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8588.454894] ? do_wait_intr+0xf0/0xf0 [ 8588.457257] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8588.473049] kthread+0x1d1/0x200 [ 8588.474964] ? set_kthread_struct+0x70/0x70 [ 8588.479353] ret_from_fork+0x1f/0x30 [ 8590.310067] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6567s! [ 8590.315707] Showing busy workqueues and worker pools: [ 8590.324716] workqueue events: flags=0x0 [ 8590.326918] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8590.331868] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8590.336570] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8590.347278] pending: sec_gc_main [ptlrpc] [ 8590.351419] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8590.359967] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8590.369613] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8590.374092] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8590.384752] workqueue events_long: flags=0x0 [ 8590.387512] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8590.391472] pending: lu_objects_flush [obdclass] [ 8590.394577] workqueue events_unbound: flags=0x2 [ 8590.401131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8590.410757] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8590.417036] workqueue events_power_efficient: flags=0x80 [ 8590.420292] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8590.424005] pending: neigh_periodic_work, do_cache_clean [ 8590.427126] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8590.431117] pending: neigh_periodic_work, check_lifetime [ 8590.434188] workqueue mm_percpu_wq: flags=0x8 [ 8590.436783] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8590.440305] pending: vmstat_update [ 8590.442672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8590.446026] pending: vmstat_update [ 8590.447937] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8590.451345] pending: vmstat_update [ 8590.453548] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8590.457150] pending: vmstat_update [ 8590.459124] workqueue writeback: flags=0x4a [ 8590.461617] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8590.465210] pending: wb_workfn, wb_workfn [ 8590.468352] workqueue xprtiod: flags=0xa [ 8590.470431] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8590.474450] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8590.479326] workqueue ipv6_addrconf: flags=0x40008 [ 8590.481906] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8590.485730] pending: addrconf_verify_work [ 8590.488361] workqueue knbd0-recv: flags=0x1a [ 8590.491106] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8590.495035] in-flight: 647:recv_work [ 8590.497082] workqueue ptlrpc_pinger: flags=0x2 [ 8590.499534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8590.503241] pending: ptlrpc_pinger_main [ptlrpc] [ 8590.506343] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8580s workers=2 idle: 652 [ 8592.061484] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8592.066153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8592.129959] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8592.148423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8592.159206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8592.166210] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8592.184169] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8592.194323] RAX: 0000000000006728 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8592.201283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8592.207613] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8592.216302] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8592.220247] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8592.223612] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8592.227049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8592.230019] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8592.233399] Call Trace: [ 8592.234851] [ 8592.235941] ? show_regs.cold.9+0x22/0x2f [ 8592.238365] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8592.241219] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8592.246148] ? watchdog+0x40/0x40 [ 8592.247890] ? hrtimer_interrupt+0x128/0x2c0 [ 8592.250364] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8592.255379] ? apic_timer_interrupt+0xf/0x20 [ 8592.258392] [ 8592.260104] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8592.264078] do_raw_spin_lock+0x177/0x190 [ 8592.266564] _raw_spin_lock+0x21/0x30 [ 8592.268022] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8592.274376] ? do_raw_spin_unlock+0x75/0x190 [ 8592.276228] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8592.279436] ? do_wait_intr+0xf0/0xf0 [ 8592.281399] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8592.285027] mdc_close+0x53b/0xd70 [mdc] [ 8592.287811] lmv_close+0x18d/0x490 [lmv] [ 8592.289959] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8592.293176] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8592.296196] ll_md_close+0x274/0x800 [lustre] [ 8592.299043] ll_file_release+0x128/0xc50 [lustre] [ 8592.301739] __fput+0x10d/0x440 [ 8592.303732] ____fput+0x12/0x20 [ 8592.305524] task_work_run+0xc6/0x110 [ 8592.310708] exit_to_usermode_loop+0x1dd/0x1f0 [ 8592.312674] do_syscall_64+0x3d6/0x440 [ 8592.317805] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8592.321691] RIP: 0033:0x7f1a63983cd8 [ 8592.324981] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8592.334261] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8592.340062] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8592.343709] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8592.349090] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8592.353037] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8592.356714] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8616.054944] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 8616.058033] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8616.058113] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 8616.058121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8616.058215] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8616.058235] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8616.058239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8616.058254] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8616.058259] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8616.058264] RAX: 0000000000007ead RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8616.058267] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8616.058269] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8616.058272] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8616.058274] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8616.058278] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8616.058282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8616.058285] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8616.058294] Call Trace: [ 8616.058300] [ 8616.058306] ? show_regs.cold.9+0x22/0x2f [ 8616.058320] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8616.058328] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8616.058333] ? watchdog+0x40/0x40 [ 8616.058337] ? hrtimer_interrupt+0x128/0x2c0 [ 8616.058342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8616.058351] ? apic_timer_interrupt+0xf/0x20 [ 8616.058367] [ 8616.058371] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8616.058378] do_raw_spin_lock+0x177/0x190 [ 8616.058382] _raw_spin_lock+0x21/0x30 [ 8616.058387] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8616.058700] ? do_raw_spin_unlock+0x75/0x190 [ 8616.058707] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8616.058917] ? do_wait_intr+0xf0/0xf0 [ 8616.058927] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8616.059128] mdc_close+0x53b/0xd70 [mdc] [ 8616.059178] lmv_close+0x18d/0x490 [lmv] [ 8616.059207] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8616.059329] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8616.059422] ll_md_close+0x274/0x800 [lustre] [ 8616.059500] ll_file_release+0x128/0xc50 [lustre] [ 8616.059576] __fput+0x10d/0x440 [ 8616.059583] ____fput+0x12/0x20 [ 8616.059587] task_work_run+0xc6/0x110 [ 8616.059594] exit_to_usermode_loop+0x1dd/0x1f0 [ 8616.059602] do_syscall_64+0x3d6/0x440 [ 8616.059608] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8616.059617] RIP: 0033:0x7f1a63983cd8 [ 8616.059626] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8616.059630] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8616.059634] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8616.059637] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8616.059639] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8616.059641] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8616.059643] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8616.061266] Modules linked in: [ 8616.063898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8616.063979] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8616.063986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8616.063989] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8616.064004] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8616.064009] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8616.064013] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8616.064015] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8616.064018] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8616.064020] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8616.064022] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8616.064025] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8616.064028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8616.064031] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8616.064040] Call Trace: [ 8616.064047] [ 8616.064053] ? show_regs.cold.9+0x22/0x2f [ 8616.064065] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8616.064072] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8616.064077] ? watchdog+0x40/0x40 [ 8616.064080] ? hrtimer_interrupt+0x128/0x2c0 [ 8616.064084] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8616.064091] ? apic_timer_interrupt+0xf/0x20 [ 8616.064096] [ 8616.064099] ? native_safe_halt+0x2e/0x50 [ 8616.064103] kvm_wait+0xcd/0xe0 [ 8616.064112] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8616.064120] do_raw_spin_lock+0x177/0x190 [ 8616.064125] _raw_spin_lock+0x21/0x30 [ 8616.064130] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8616.064529] obd_mod_free+0x46/0x140 [ptlrpc] [ 8616.064726] mdc_commit_open+0x91/0xb0 [mdc] [ 8616.064768] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8616.064965] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8616.065163] after_reply+0xe2c/0x1780 [ptlrpc] [ 8616.065447] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8616.065655] ? __next_timer_interrupt+0x160/0x160 [ 8616.065667] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8616.066143] ? do_wait_intr+0xf0/0xf0 [ 8616.066154] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8616.075613] lustre(O) [ 8616.091009] kthread+0x1d1/0x200 [ 8616.097149] osp(O) [ 8616.100532] ? set_kthread_struct+0x70/0x70 [ 8616.100552] ret_from_fork+0x1f/0x30 [ 8616.491183] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8616.514961] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8616.521631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8616.524136] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8616.528182] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8616.541150] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8616.545166] RAX: 0000000000007c09 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8616.549832] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8616.555495] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8616.557608] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8616.560298] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8616.564067] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8616.568009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8616.570961] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8616.573998] Call Trace: [ 8616.575186] [ 8616.576123] ? show_regs.cold.9+0x22/0x2f [ 8616.577727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8616.579924] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8616.581675] ? watchdog+0x40/0x40 [ 8616.583629] ? hrtimer_interrupt+0x128/0x2c0 [ 8616.584960] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8616.586767] ? apic_timer_interrupt+0xf/0x20 [ 8616.588718] [ 8616.589619] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8616.591745] do_raw_spin_lock+0x177/0x190 [ 8616.594070] _raw_spin_lock+0x21/0x30 [ 8616.595174] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8616.596958] ? do_raw_spin_unlock+0x75/0x190 [ 8616.598170] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8616.600029] ? do_wait_intr+0xf0/0xf0 [ 8616.601240] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8616.602784] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8616.604486] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8616.606165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8616.607830] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8616.609336] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8616.610561] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8616.611749] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8616.613184] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8616.615359] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8616.616911] lmv_intent_open+0x338/0x1150 [lmv] [ 8616.618183] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8616.620018] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8616.621644] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8616.622818] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8616.624070] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8616.625625] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8616.627485] ? do_raw_spin_unlock+0x75/0x190 [ 8616.628625] ? _raw_spin_unlock+0x12/0x30 [ 8616.629661] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8616.630965] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8616.632311] ll_atomic_open+0x36a/0xd70 [lustre] [ 8616.633981] lookup_open+0x356/0xb40 [ 8616.635099] do_last+0x313/0xfc0 [ 8616.635940] ? nd_jump_root+0xe5/0x160 [ 8616.642513] ? path_init+0x437/0x520 [ 8616.644677] path_openat+0xf7/0x500 [ 8616.646049] ? do_raw_spin_unlock+0x75/0x190 [ 8616.649420] do_filp_open+0x99/0x140 [ 8616.652226] ? getname_flags+0x6e/0x330 [ 8616.656443] ? __check_object_size+0xff/0x256 [ 8616.657771] ? do_raw_spin_unlock+0x75/0x190 [ 8616.659080] ? _raw_spin_unlock+0x12/0x30 [ 8616.661128] do_sys_openat2+0x2b4/0x410 [ 8616.662984] do_sys_open+0x73/0xa0 [ 8616.664522] __x64_sys_openat+0x24/0x30 [ 8616.666152] do_syscall_64+0xc1/0x440 [ 8616.668047] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8616.670562] RIP: 0033:0x7f1a63983332 [ 8616.672795] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8616.681176] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8616.685076] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8616.688780] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8616.692002] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8616.694401] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8616.697715] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8620.057029] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 8620.059857] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8620.100194] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8620.112415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8620.125429] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8620.129343] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8620.146736] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8620.152776] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8620.157476] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8620.162148] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8620.168228] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8620.171505] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8620.175481] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8620.180245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8620.184023] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8620.192181] Call Trace: [ 8620.195541] [ 8620.197415] ? show_regs.cold.9+0x22/0x2f [ 8620.200684] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8620.203416] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8620.205029] ? watchdog+0x40/0x40 [ 8620.208757] ? hrtimer_interrupt+0x128/0x2c0 [ 8620.211450] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8620.214389] ? apic_timer_interrupt+0xf/0x20 [ 8620.217493] [ 8620.219482] ? native_safe_halt+0x2e/0x50 [ 8620.222700] kvm_wait+0xcd/0xe0 [ 8620.224061] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8620.229196] do_raw_spin_lock+0x177/0x190 [ 8620.234648] _raw_spin_lock+0x21/0x30 [ 8620.241302] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8620.243491] ? do_raw_spin_unlock+0x75/0x190 [ 8620.245731] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8620.248055] ? do_wait_intr+0xf0/0xf0 [ 8620.250547] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8620.253752] mdc_close+0x53b/0xd70 [mdc] [ 8620.255667] lmv_close+0x18d/0x490 [lmv] [ 8620.257715] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8620.265630] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8620.275909] ll_md_close+0x274/0x800 [lustre] [ 8620.281840] ll_file_release+0x128/0xc50 [lustre] [ 8620.289759] __fput+0x10d/0x440 [ 8620.291901] ____fput+0x12/0x20 [ 8620.292890] task_work_run+0xc6/0x110 [ 8620.293951] exit_to_usermode_loop+0x1dd/0x1f0 [ 8620.300792] do_syscall_64+0x3d6/0x440 [ 8620.311507] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8620.320260] RIP: 0033:0x7f1a63983cd8 [ 8620.322193] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8620.353677] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8620.370229] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8620.380751] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8620.390629] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8620.396380] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8620.405510] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8620.513057] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6597s! [ 8620.518694] Showing busy workqueues and worker pools: [ 8620.520801] workqueue events: flags=0x0 [ 8620.523122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8620.526522] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8620.532717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8620.535723] pending: sec_gc_main [ptlrpc] [ 8620.538011] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8620.544834] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8620.551022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8620.564584] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8620.568386] workqueue events_long: flags=0x0 [ 8620.570545] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8620.576267] pending: lu_objects_flush [obdclass] [ 8620.581700] workqueue events_unbound: flags=0x2 [ 8620.587818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8620.598171] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8620.604685] workqueue events_power_efficient: flags=0x80 [ 8620.606832] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8620.609571] pending: neigh_periodic_work, do_cache_clean [ 8620.612263] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8620.616606] pending: neigh_periodic_work, check_lifetime [ 8620.621402] workqueue mm_percpu_wq: flags=0x8 [ 8620.624263] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8620.630100] pending: vmstat_update [ 8620.633035] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8620.635734] pending: vmstat_update [ 8620.638140] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8620.643710] pending: vmstat_update [ 8620.646193] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8620.649156] pending: vmstat_update [ 8620.651037] workqueue writeback: flags=0x4a [ 8620.653217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8620.656423] pending: wb_workfn, wb_workfn [ 8620.659955] workqueue xprtiod: flags=0xa [ 8620.664318] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8620.667161] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8620.671022] workqueue ipv6_addrconf: flags=0x40008 [ 8620.676931] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8620.680606] pending: addrconf_verify_work [ 8620.682551] workqueue knbd0-recv: flags=0x1a [ 8620.685130] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8620.688280] in-flight: 647:recv_work [ 8620.690219] workqueue ptlrpc_pinger: flags=0x2 [ 8620.695079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8620.699567] pending: ptlrpc_pinger_main [ptlrpc] [ 8620.703171] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8610s workers=2 idle: 652 [ 8644.056034] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8644.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8644.059234] Modules linked in: lustre(O) [ 8644.065452] Modules linked in: [ 8644.065456] osp(O) ofd(O) [ 8644.068410] lustre(O) [ 8644.069744] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8644.069817] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8644.069826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8644.069829] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8644.069846] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8644.069851] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8644.069856] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8644.069858] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8644.069860] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8644.069862] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8644.069865] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8644.069869] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8644.069872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8644.069875] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8644.069884] Call Trace: [ 8644.069889] [ 8644.069895] ? show_regs.cold.9+0x22/0x2f [ 8644.069910] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8644.069916] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8644.069920] ? watchdog+0x40/0x40 [ 8644.069925] ? hrtimer_interrupt+0x128/0x2c0 [ 8644.069929] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8644.069937] ? apic_timer_interrupt+0xf/0x20 [ 8644.069942] [ 8644.069945] ? native_safe_halt+0x2e/0x50 [ 8644.069963] kvm_wait+0xcd/0xe0 [ 8644.069971] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8644.071311] osp(O) [ 8644.072568] do_raw_spin_lock+0x177/0x190 [ 8644.072585] _raw_spin_lock+0x21/0x30 [ 8644.072592] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8644.072934] ? do_raw_spin_unlock+0x75/0x190 [ 8644.072940] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8644.104395] ofd(O) [ 8644.109314] ? do_wait_intr+0xf0/0xf0 [ 8644.116803] lod(O) [ 8644.119117] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8644.136317] mdt(O) [ 8644.140479] mdc_close+0x53b/0xd70 [mdc] [ 8644.143977] mdd(O) [ 8644.146537] lmv_close+0x18d/0x490 [lmv] [ 8644.151709] mgs(O) lquota(O) lfsck(O) obdecho(O) [ 8644.154368] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8644.160225] mgc(O) [ 8644.163326] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8644.163449] ll_md_close+0x274/0x800 [lustre] [ 8644.163526] ll_file_release+0x128/0xc50 [lustre] [ 8644.163602] __fput+0x10d/0x440 [ 8644.163611] ____fput+0x12/0x20 [ 8644.163615] task_work_run+0xc6/0x110 [ 8644.163622] exit_to_usermode_loop+0x1dd/0x1f0 [ 8644.163629] do_syscall_64+0x3d6/0x440 [ 8644.163632] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8644.163640] RIP: 0033:0x7f1a63983cd8 [ 8644.163647] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8644.163651] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8644.163660] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8644.163663] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8644.163665] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8644.163667] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8644.163670] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8644.289531] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8644.305791] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8644.311959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8644.316768] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8644.318762] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8644.341607] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8644.344256] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8644.350981] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8644.357712] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8644.362873] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8644.366955] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8644.370189] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8644.373550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8644.375208] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8644.379114] Call Trace: [ 8644.380011] [ 8644.380701] ? show_regs.cold.9+0x22/0x2f [ 8644.383057] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8644.388939] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8644.391728] ? watchdog+0x40/0x40 [ 8644.399307] ? hrtimer_interrupt+0x128/0x2c0 [ 8644.401837] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8644.406514] ? apic_timer_interrupt+0xf/0x20 [ 8644.408743] [ 8644.409689] ? native_safe_halt+0x2e/0x50 [ 8644.415954] kvm_wait+0xcd/0xe0 [ 8644.418135] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8644.420874] do_raw_spin_lock+0x177/0x190 [ 8644.426843] _raw_spin_lock+0x21/0x30 [ 8644.429857] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8644.433869] obd_mod_free+0x46/0x140 [ptlrpc] [ 8644.436317] mdc_commit_open+0x91/0xb0 [mdc] [ 8644.438366] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8644.443122] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8644.449651] after_reply+0xe2c/0x1780 [ptlrpc] [ 8644.454361] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8644.458404] ? __next_timer_interrupt+0x160/0x160 [ 8644.469795] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8644.473586] ? do_wait_intr+0xf0/0xf0 [ 8644.476406] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8644.479264] kthread+0x1d1/0x200 [ 8644.481162] ? set_kthread_struct+0x70/0x70 [ 8644.483247] ret_from_fork+0x1f/0x30 [ 8648.053122] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 8648.057025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 8648.058976] Modules linked in: [ 8648.063322] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8648.063406] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8648.063414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8648.063417] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8648.063430] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8648.063433] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8648.063438] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8648.063440] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8648.063443] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8648.063445] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8648.063447] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8648.063451] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8648.063453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8648.063455] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8648.063464] Call Trace: [ 8648.063476] [ 8648.063482] ? show_regs.cold.9+0x22/0x2f [ 8648.063492] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8648.063503] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8648.063507] ? watchdog+0x40/0x40 [ 8648.063512] ? hrtimer_interrupt+0x128/0x2c0 [ 8648.063516] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8648.063523] ? apic_timer_interrupt+0xf/0x20 [ 8648.063531] [ 8648.063534] ? native_safe_halt+0x2e/0x50 [ 8648.063538] kvm_wait+0xcd/0xe0 [ 8648.063546] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8648.063554] do_raw_spin_lock+0x177/0x190 [ 8648.063558] _raw_spin_lock+0x21/0x30 [ 8648.063562] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8648.064154] ? do_raw_spin_unlock+0x75/0x190 [ 8648.064160] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8648.064767] ? do_wait_intr+0xf0/0xf0 [ 8648.064775] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8648.070141] lustre(O) osp(O) ofd(O) lod(O) [ 8648.093549] mdc_close+0x53b/0xd70 [mdc] [ 8648.093595] lmv_close+0x18d/0x490 [lmv] [ 8648.093626] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8648.093734] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8648.093870] ll_md_close+0x274/0x800 [lustre] [ 8648.094022] ll_file_release+0x128/0xc50 [lustre] [ 8648.094174] __fput+0x10d/0x440 [ 8648.094184] ____fput+0x12/0x20 [ 8648.094188] task_work_run+0xc6/0x110 [ 8648.094196] exit_to_usermode_loop+0x1dd/0x1f0 [ 8648.094204] do_syscall_64+0x3d6/0x440 [ 8648.094209] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8648.094217] RIP: 0033:0x7f1a63983cd8 [ 8648.094225] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8648.094229] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8648.094234] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8648.094236] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8648.094238] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8648.094243] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8648.094245] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8648.262715] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8648.283828] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8648.290141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8648.295148] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8648.297523] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8648.306893] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8648.310651] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8648.313852] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8648.317490] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8648.321629] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8648.325563] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8648.329373] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8648.334182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8648.337422] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8648.341652] Call Trace: [ 8648.342839] [ 8648.343826] ? show_regs.cold.9+0x22/0x2f [ 8648.346333] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8648.348687] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8648.351298] ? watchdog+0x40/0x40 [ 8648.353618] ? hrtimer_interrupt+0x128/0x2c0 [ 8648.356212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8648.359570] ? apic_timer_interrupt+0xf/0x20 [ 8648.362368] [ 8648.363535] ? native_safe_halt+0x2e/0x50 [ 8648.365632] kvm_wait+0xcd/0xe0 [ 8648.367372] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8648.370331] do_raw_spin_lock+0x177/0x190 [ 8648.372424] _raw_spin_lock+0x21/0x30 [ 8648.374795] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8648.377571] ? do_raw_spin_unlock+0x75/0x190 [ 8648.379618] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8648.382523] ? do_wait_intr+0xf0/0xf0 [ 8648.384296] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8648.387707] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8648.390678] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8648.393678] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8648.396418] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8648.399324] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8648.401472] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8648.404438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8648.406984] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8648.410418] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8648.413582] lmv_intent_open+0x338/0x1150 [lmv] [ 8648.415739] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8648.418357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8648.421652] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8648.425217] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8648.428554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8648.431226] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8648.433824] ? do_raw_spin_unlock+0x75/0x190 [ 8648.436263] ? _raw_spin_unlock+0x12/0x30 [ 8648.438479] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8648.440640] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8648.443485] ll_atomic_open+0x36a/0xd70 [lustre] [ 8648.446046] lookup_open+0x356/0xb40 [ 8648.447729] do_last+0x313/0xfc0 [ 8648.449210] ? nd_jump_root+0xe5/0x160 [ 8648.450710] ? path_init+0x437/0x520 [ 8648.452731] path_openat+0xf7/0x500 [ 8648.454423] ? do_raw_spin_unlock+0x75/0x190 [ 8648.456415] do_filp_open+0x99/0x140 [ 8648.457804] ? getname_flags+0x6e/0x330 [ 8648.459386] ? __check_object_size+0xff/0x256 [ 8648.461383] ? do_raw_spin_unlock+0x75/0x190 [ 8648.463440] ? _raw_spin_unlock+0x12/0x30 [ 8648.465434] do_sys_openat2+0x2b4/0x410 [ 8648.466996] do_sys_open+0x73/0xa0 [ 8648.468671] __x64_sys_openat+0x24/0x30 [ 8648.470449] do_syscall_64+0xc1/0x440 [ 8648.472092] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8648.474375] RIP: 0033:0x7f1a63983332 [ 8648.475837] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8648.484026] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8648.487772] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8648.490861] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8648.493819] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8648.496642] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8648.499423] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8650.721771] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6627s! [ 8650.733473] Showing busy workqueues and worker pools: [ 8650.739992] workqueue events: flags=0x0 [ 8650.741644] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8650.748180] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8650.757673] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8650.761745] pending: sec_gc_main [ptlrpc] [ 8650.772373] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8650.779706] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8650.793426] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8650.797921] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8650.808314] workqueue events_long: flags=0x0 [ 8650.810670] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8650.814831] pending: lu_objects_flush [obdclass] [ 8650.818751] workqueue events_unbound: flags=0x2 [ 8650.821322] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8650.826010] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8650.835381] workqueue events_power_efficient: flags=0x80 [ 8650.839884] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8650.843683] pending: neigh_periodic_work, do_cache_clean [ 8650.846934] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8650.858318] pending: neigh_periodic_work, check_lifetime [ 8650.865557] workqueue mm_percpu_wq: flags=0x8 [ 8650.867230] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8650.872619] pending: vmstat_update [ 8650.876938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8650.885852] pending: vmstat_update [ 8650.889807] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8650.895606] pending: vmstat_update [ 8650.901462] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8650.908356] pending: vmstat_update [ 8650.911125] workqueue writeback: flags=0x4a [ 8650.912804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8650.919952] pending: wb_workfn, wb_workfn [ 8650.926125] workqueue xprtiod: flags=0xa [ 8650.930143] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8650.938453] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8650.944054] workqueue ipv6_addrconf: flags=0x40008 [ 8650.947905] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8650.951873] pending: addrconf_verify_work [ 8650.954227] workqueue knbd0-recv: flags=0x1a [ 8650.957768] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8650.968359] in-flight: 647:recv_work [ 8650.975661] workqueue ptlrpc_pinger: flags=0x2 [ 8650.981508] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8650.991301] pending: ptlrpc_pinger_main [ptlrpc] [ 8650.999150] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8641s workers=2 idle: 652 [ 8672.056024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8672.058022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8672.058788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8672.058874] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8672.058880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8672.058883] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8672.058895] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8672.058899] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8672.058903] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8672.058906] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8672.058908] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8672.058910] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8672.058912] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8672.058915] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8672.058918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8672.058921] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8672.058928] Call Trace: [ 8672.058932] [ 8672.058937] ? show_regs.cold.9+0x22/0x2f [ 8672.058946] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8672.058952] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8672.058956] ? watchdog+0x40/0x40 [ 8672.058960] ? hrtimer_interrupt+0x128/0x2c0 [ 8672.058963] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8672.058969] ? apic_timer_interrupt+0xf/0x20 [ 8672.058973] [ 8672.058976] ? native_safe_halt+0x2e/0x50 [ 8672.058980] kvm_wait+0xcd/0xe0 [ 8672.058987] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8672.058995] do_raw_spin_lock+0x177/0x190 [ 8672.058999] _raw_spin_lock+0x21/0x30 [ 8672.065140] Modules linked in: lustre(O) [ 8672.117502] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8672.121967] osp(O) [ 8672.130205] ? do_raw_spin_unlock+0x75/0x190 [ 8672.131755] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8672.131835] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8672.131841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8672.131844] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8672.131855] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8672.131860] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8672.131864] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8672.131866] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8672.131869] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8672.131871] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8672.131873] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8672.131877] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8672.131879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8672.131881] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8672.131889] Call Trace: [ 8672.131893] [ 8672.131897] ? show_regs.cold.9+0x22/0x2f [ 8672.131907] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8672.131913] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8672.131918] ? watchdog+0x40/0x40 [ 8672.131922] ? hrtimer_interrupt+0x128/0x2c0 [ 8672.131926] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8672.131932] ? apic_timer_interrupt+0xf/0x20 [ 8672.131937] [ 8672.131941] ? native_safe_halt+0x2e/0x50 [ 8672.131945] kvm_wait+0xcd/0xe0 [ 8672.131953] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8672.131961] do_raw_spin_lock+0x177/0x190 [ 8672.131965] _raw_spin_lock+0x21/0x30 [ 8672.131970] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8672.132236] obd_mod_free+0x46/0x140 [ptlrpc] [ 8672.132493] mdc_commit_open+0x91/0xb0 [mdc] [ 8672.132533] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8672.132827] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8672.133060] after_reply+0xe2c/0x1780 [ptlrpc] [ 8672.133257] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8672.133501] ? __next_timer_interrupt+0x160/0x160 [ 8672.133510] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8672.133763] ? do_wait_intr+0xf0/0xf0 [ 8672.133771] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8672.133977] kthread+0x1d1/0x200 [ 8672.133982] ? set_kthread_struct+0x70/0x70 [ 8672.133986] ret_from_fork+0x1f/0x30 [ 8672.547672] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8672.551505] ? do_wait_intr+0xf0/0xf0 [ 8672.557115] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8672.562018] mdc_close+0x53b/0xd70 [mdc] [ 8672.566108] lmv_close+0x18d/0x490 [lmv] [ 8672.571316] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8672.581041] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8672.587207] ll_md_close+0x274/0x800 [lustre] [ 8672.597085] ll_file_release+0x128/0xc50 [lustre] [ 8672.610756] __fput+0x10d/0x440 [ 8672.612002] ____fput+0x12/0x20 [ 8672.613216] task_work_run+0xc6/0x110 [ 8672.616556] exit_to_usermode_loop+0x1dd/0x1f0 [ 8672.621710] do_syscall_64+0x3d6/0x440 [ 8672.626282] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8672.630086] RIP: 0033:0x7f1a63983cd8 [ 8672.633478] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8672.648587] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8672.651835] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8672.655037] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8672.662582] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8672.666011] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8672.668742] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8676.053025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 8676.055816] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8676.057149] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 8676.092643] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8676.092659] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8676.092663] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8676.095174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8676.095250] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8676.095257] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8676.095260] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8676.095272] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8676.095276] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8676.095280] RAX: 000000000000733d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8676.095283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8676.095285] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8676.095287] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8676.095290] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8676.095293] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8676.095296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8676.095298] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8676.095305] Call Trace: [ 8676.095309] [ 8676.095314] ? show_regs.cold.9+0x22/0x2f [ 8676.095322] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8676.095328] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8676.095332] ? watchdog+0x40/0x40 [ 8676.095336] ? hrtimer_interrupt+0x128/0x2c0 [ 8676.095339] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8676.095347] ? apic_timer_interrupt+0xf/0x20 [ 8676.095351] [ 8676.095353] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8676.095359] do_raw_spin_lock+0x177/0x190 [ 8676.095362] _raw_spin_lock+0x21/0x30 [ 8676.095367] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8676.095753] ? do_raw_spin_unlock+0x75/0x190 [ 8676.095765] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8676.107448] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8676.110538] ? do_wait_intr+0xf0/0xf0 [ 8676.110564] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8676.111104] mdc_close+0x53b/0xd70 [mdc] [ 8676.111144] lmv_close+0x18d/0x490 [lmv] [ 8676.111173] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8676.111417] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8676.111497] ll_md_close+0x274/0x800 [lustre] [ 8676.111579] ll_file_release+0x128/0xc50 [lustre] [ 8676.111665] __fput+0x10d/0x440 [ 8676.111675] ____fput+0x12/0x20 [ 8676.111679] task_work_run+0xc6/0x110 [ 8676.111687] exit_to_usermode_loop+0x1dd/0x1f0 [ 8676.111695] do_syscall_64+0x3d6/0x440 [ 8676.111703] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8676.111715] RIP: 0033:0x7f1a63983cd8 [ 8676.111724] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8676.111728] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8676.111740] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8676.111742] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8676.111745] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8676.111747] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8676.111749] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8676.527844] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8676.534706] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8676.538599] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8676.545706] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8676.551311] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8676.557438] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8676.559340] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8676.576161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8676.584825] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8676.590115] Call Trace: [ 8676.591035] [ 8676.593020] ? show_regs.cold.9+0x22/0x2f [ 8676.599339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8676.606038] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8676.611483] ? watchdog+0x40/0x40 [ 8676.616004] ? hrtimer_interrupt+0x128/0x2c0 [ 8676.618339] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8676.621644] ? apic_timer_interrupt+0xf/0x20 [ 8676.625315] [ 8676.626145] ? native_safe_halt+0x2e/0x50 [ 8676.627431] kvm_wait+0xcd/0xe0 [ 8676.628397] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8676.635881] do_raw_spin_lock+0x177/0x190 [ 8676.638142] _raw_spin_lock+0x21/0x30 [ 8676.641597] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8676.648006] ? do_raw_spin_unlock+0x75/0x190 [ 8676.652897] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8676.655662] ? do_wait_intr+0xf0/0xf0 [ 8676.658596] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8676.663539] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8676.667933] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8676.671464] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8676.674311] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8676.680667] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8676.684494] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8676.685884] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8676.696508] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8676.699648] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8676.705823] lmv_intent_open+0x338/0x1150 [lmv] [ 8676.713752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8676.728259] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8676.732336] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8676.733733] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8676.736516] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8676.740413] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8676.744012] ? do_raw_spin_unlock+0x75/0x190 [ 8676.745628] ? _raw_spin_unlock+0x12/0x30 [ 8676.756675] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8676.759735] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8676.761719] ll_atomic_open+0x36a/0xd70 [lustre] [ 8676.778357] lookup_open+0x356/0xb40 [ 8676.792077] do_last+0x313/0xfc0 [ 8676.793120] ? nd_jump_root+0xe5/0x160 [ 8676.798075] ? path_init+0x437/0x520 [ 8676.799731] path_openat+0xf7/0x500 [ 8676.801007] ? do_raw_spin_unlock+0x75/0x190 [ 8676.812505] do_filp_open+0x99/0x140 [ 8676.820739] ? getname_flags+0x6e/0x330 [ 8676.824699] ? __check_object_size+0xff/0x256 [ 8676.836255] ? do_raw_spin_unlock+0x75/0x190 [ 8676.838904] ? _raw_spin_unlock+0x12/0x30 [ 8676.840639] do_sys_openat2+0x2b4/0x410 [ 8676.844624] do_sys_open+0x73/0xa0 [ 8676.845943] __x64_sys_openat+0x24/0x30 [ 8676.848317] do_syscall_64+0xc1/0x440 [ 8676.849467] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8676.857022] RIP: 0033:0x7f1a63983332 [ 8676.860027] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8676.878731] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8676.882508] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8676.889157] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8676.900048] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8676.906355] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8676.910092] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8681.445986] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6658s! [ 8681.469035] Showing busy workqueues and worker pools: [ 8681.471898] workqueue events: flags=0x0 [ 8681.483514] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8681.493189] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8681.500222] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8681.509892] pending: sec_gc_main [ptlrpc] [ 8681.516028] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8681.525012] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8681.538374] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8681.551582] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8681.560140] workqueue events_long: flags=0x0 [ 8681.566910] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8681.576012] pending: lu_objects_flush [obdclass] [ 8681.581245] workqueue events_unbound: flags=0x2 [ 8681.590185] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8681.599320] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8681.609193] workqueue events_power_efficient: flags=0x80 [ 8681.611113] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8681.624844] pending: neigh_periodic_work, do_cache_clean [ 8681.630663] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8681.636974] pending: neigh_periodic_work, check_lifetime [ 8681.639263] workqueue mm_percpu_wq: flags=0x8 [ 8681.645473] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8681.653535] pending: vmstat_update [ 8681.656529] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8681.670099] pending: vmstat_update [ 8681.674156] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8681.684999] pending: vmstat_update [ 8681.689790] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8681.695996] pending: vmstat_update [ 8681.697379] workqueue writeback: flags=0x4a [ 8681.706358] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8681.718227] pending: wb_workfn, wb_workfn [ 8681.726373] workqueue xprtiod: flags=0xa [ 8681.734179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8681.744136] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8681.753043] workqueue ipv6_addrconf: flags=0x40008 [ 8681.754644] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8681.770435] pending: addrconf_verify_work [ 8681.783170] workqueue knbd0-recv: flags=0x1a [ 8681.792759] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8681.800407] in-flight: 647:recv_work [ 8681.804490] workqueue ptlrpc_pinger: flags=0x2 [ 8681.808181] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8681.816910] pending: ptlrpc_pinger_main [ptlrpc] [ 8681.825063] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8671s workers=2 idle: 652 [ 8700.056034] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8700.059365] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8700.059901] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8700.059974] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8700.059980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8700.059983] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8700.059997] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8700.060001] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8700.060006] RAX: 0000000000007974 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8700.060009] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8700.060011] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8700.060013] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8700.060015] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8700.060018] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8700.060022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8700.060025] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8700.060034] Call Trace: [ 8700.060040] [ 8700.060046] ? show_regs.cold.9+0x22/0x2f [ 8700.060062] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8700.060069] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8700.060074] ? watchdog+0x40/0x40 [ 8700.060078] ? hrtimer_interrupt+0x128/0x2c0 [ 8700.060082] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8700.060090] ? apic_timer_interrupt+0xf/0x20 [ 8700.060096] [ 8700.060098] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8700.060105] do_raw_spin_lock+0x177/0x190 [ 8700.060110] _raw_spin_lock+0x21/0x30 [ 8700.060115] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8700.060741] ? do_raw_spin_unlock+0x75/0x190 [ 8700.060747] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8700.061030] ? do_wait_intr+0xf0/0xf0 [ 8700.061038] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8700.061504] mdc_close+0x53b/0xd70 [mdc] [ 8700.061561] lmv_close+0x18d/0x490 [lmv] [ 8700.061589] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8700.061698] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8700.068514] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8700.068621] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8700.068629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8700.068645] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8700.068660] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8700.068665] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8700.068673] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8700.068676] RDX: 000000000000150c RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8700.068678] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8700.068681] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8700.068683] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8700.068687] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8700.068691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8700.068693] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8700.068701] Call Trace: [ 8700.068707] [ 8700.068714] ? show_regs.cold.9+0x22/0x2f [ 8700.068724] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8700.068736] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8700.068741] ? watchdog+0x40/0x40 [ 8700.068746] ? hrtimer_interrupt+0x128/0x2c0 [ 8700.068751] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8700.068759] ? apic_timer_interrupt+0xf/0x20 [ 8700.068768] [ 8700.068771] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8700.068775] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8700.068780] do_raw_spin_lock+0x177/0x190 [ 8700.068784] _raw_spin_lock+0x21/0x30 [ 8700.068790] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8700.069170] obd_mod_free+0x46/0x140 [ptlrpc] [ 8700.100657] ll_md_close+0x274/0x800 [lustre] [ 8700.111068] mdc_commit_open+0x91/0xb0 [mdc] [ 8700.111123] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8700.112525] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8700.122834] ll_file_release+0x128/0xc50 [lustre] [ 8700.123005] __fput+0x10d/0x440 [ 8700.123013] ____fput+0x12/0x20 [ 8700.123017] task_work_run+0xc6/0x110 [ 8700.123024] exit_to_usermode_loop+0x1dd/0x1f0 [ 8700.123029] do_syscall_64+0x3d6/0x440 [ 8700.123032] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8700.123039] RIP: 0033:0x7f1a63983cd8 [ 8700.123045] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8700.123048] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8700.123052] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8700.123054] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8700.123055] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8700.123057] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8700.123058] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8700.500794] after_reply+0xe2c/0x1780 [ptlrpc] [ 8700.504449] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8700.510453] ? __next_timer_interrupt+0x160/0x160 [ 8700.514199] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8700.516691] ? do_wait_intr+0xf0/0xf0 [ 8700.518140] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8700.520711] kthread+0x1d1/0x200 [ 8700.522878] ? set_kthread_struct+0x70/0x70 [ 8700.528035] ret_from_fork+0x1f/0x30 [ 8704.057130] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 8704.063149] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8704.087441] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8704.094613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8704.098777] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8704.102983] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8704.123170] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8704.129892] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8704.136141] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8704.140245] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8704.150053] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8704.156964] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8704.160988] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8704.163961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8704.166952] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8704.177061] Call Trace: [ 8704.178380] [ 8704.180461] ? show_regs.cold.9+0x22/0x2f [ 8704.183159] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8704.186208] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8704.189030] ? watchdog+0x40/0x40 [ 8704.193446] ? hrtimer_interrupt+0x128/0x2c0 [ 8704.196929] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8704.201868] ? apic_timer_interrupt+0xf/0x20 [ 8704.205919] [ 8704.208055] ? native_safe_halt+0x2e/0x50 [ 8704.212769] kvm_wait+0xcd/0xe0 [ 8704.214865] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8704.219587] do_raw_spin_lock+0x177/0x190 [ 8704.222227] _raw_spin_lock+0x21/0x30 [ 8704.224438] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8704.227375] ? do_raw_spin_unlock+0x75/0x190 [ 8704.229124] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8704.232552] ? do_wait_intr+0xf0/0xf0 [ 8704.235117] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8704.243118] mdc_close+0x53b/0xd70 [mdc] [ 8704.248255] lmv_close+0x18d/0x490 [lmv] [ 8704.251734] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8704.254810] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8704.267927] ll_md_close+0x274/0x800 [lustre] [ 8704.276939] ll_file_release+0x128/0xc50 [lustre] [ 8704.281434] __fput+0x10d/0x440 [ 8704.285024] ____fput+0x12/0x20 [ 8704.287013] task_work_run+0xc6/0x110 [ 8704.293349] exit_to_usermode_loop+0x1dd/0x1f0 [ 8704.298936] do_syscall_64+0x3d6/0x440 [ 8704.302900] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8704.309626] RIP: 0033:0x7f1a63983cd8 [ 8704.313276] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8704.331898] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8704.341750] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8704.350124] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8704.355997] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8704.368656] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8704.384134] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8708.053211] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 8708.062719] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8708.106221] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8708.117498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8708.127856] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8708.134035] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8708.153514] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8708.162697] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8708.169697] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8708.177094] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8708.185063] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8708.190657] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8708.193833] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8708.204249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8708.211218] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8708.217010] Call Trace: [ 8708.220147] [ 8708.221294] ? show_regs.cold.9+0x22/0x2f [ 8708.222943] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8708.224764] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8708.227689] ? watchdog+0x40/0x40 [ 8708.229374] ? hrtimer_interrupt+0x128/0x2c0 [ 8708.231742] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8708.234626] ? apic_timer_interrupt+0xf/0x20 [ 8708.238839] [ 8708.239991] ? native_safe_halt+0x2e/0x50 [ 8708.242079] kvm_wait+0xcd/0xe0 [ 8708.243607] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8708.248620] do_raw_spin_lock+0x177/0x190 [ 8708.251029] _raw_spin_lock+0x21/0x30 [ 8708.253248] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8708.259219] ? do_raw_spin_unlock+0x75/0x190 [ 8708.261677] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8708.264965] ? do_wait_intr+0xf0/0xf0 [ 8708.268190] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8708.273614] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8708.279538] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8708.284797] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8708.286802] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8708.292053] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8708.297989] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8708.300063] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8708.305697] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8708.308879] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8708.313098] lmv_intent_open+0x338/0x1150 [lmv] [ 8708.315432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8708.317930] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8708.322029] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8708.326743] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8708.332148] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8708.338155] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8708.343123] ? do_raw_spin_unlock+0x75/0x190 [ 8708.346867] ? _raw_spin_unlock+0x12/0x30 [ 8708.350004] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8708.353159] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8708.357967] ll_atomic_open+0x36a/0xd70 [lustre] [ 8708.362882] lookup_open+0x356/0xb40 [ 8708.366703] do_last+0x313/0xfc0 [ 8708.370312] ? nd_jump_root+0xe5/0x160 [ 8708.374213] ? path_init+0x437/0x520 [ 8708.376824] path_openat+0xf7/0x500 [ 8708.378996] ? do_raw_spin_unlock+0x75/0x190 [ 8708.383936] do_filp_open+0x99/0x140 [ 8708.387220] ? getname_flags+0x6e/0x330 [ 8708.388944] ? __check_object_size+0xff/0x256 [ 8708.392675] ? do_raw_spin_unlock+0x75/0x190 [ 8708.396842] ? _raw_spin_unlock+0x12/0x30 [ 8708.400244] do_sys_openat2+0x2b4/0x410 [ 8708.405594] do_sys_open+0x73/0xa0 [ 8708.409663] __x64_sys_openat+0x24/0x30 [ 8708.411021] do_syscall_64+0xc1/0x440 [ 8708.415622] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8708.421086] RIP: 0033:0x7f1a63983332 [ 8708.425436] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8708.437944] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8708.445811] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8708.452991] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8708.461192] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8708.469676] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8708.476737] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8712.168539] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6689s! [ 8712.176963] Showing busy workqueues and worker pools: [ 8712.183062] workqueue events: flags=0x0 [ 8712.188045] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8712.191504] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8712.197031] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8712.199930] pending: sec_gc_main [ptlrpc] [ 8712.203898] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8712.208056] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8712.214655] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8712.219395] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8712.225540] workqueue events_long: flags=0x0 [ 8712.227358] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8712.230037] pending: lu_objects_flush [obdclass] [ 8712.232596] workqueue events_unbound: flags=0x2 [ 8712.235592] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8712.243212] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8712.252248] workqueue events_power_efficient: flags=0x80 [ 8712.256971] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8712.268671] pending: neigh_periodic_work, do_cache_clean [ 8712.272727] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8712.279579] pending: neigh_periodic_work, check_lifetime [ 8712.289980] workqueue mm_percpu_wq: flags=0x8 [ 8712.299959] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8712.305881] pending: vmstat_update [ 8712.311672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8712.314118] pending: vmstat_update [ 8712.320989] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8712.326074] pending: vmstat_update [ 8712.329844] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8712.337807] pending: vmstat_update [ 8712.341991] workqueue writeback: flags=0x4a [ 8712.348854] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8712.356281] pending: wb_workfn, wb_workfn [ 8712.362189] workqueue xprtiod: flags=0xa [ 8712.367163] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8712.371619] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8712.375716] workqueue ipv6_addrconf: flags=0x40008 [ 8712.378176] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8712.380958] pending: addrconf_verify_work [ 8712.382735] workqueue knbd0-recv: flags=0x1a [ 8712.384881] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8712.387896] in-flight: 647:recv_work [ 8712.393260] workqueue ptlrpc_pinger: flags=0x2 [ 8712.397264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8712.405308] pending: ptlrpc_pinger_main [ptlrpc] [ 8712.408358] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8702s workers=2 idle: 652 [ 8728.057496] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8728.062883] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8728.064357] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8728.080805] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8728.080819] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8728.080821] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8728.080834] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8728.080837] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8728.080841] RAX: 0000000000000584 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8728.080843] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8728.080845] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8728.080846] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8728.080848] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8728.080850] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8728.080852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8728.080854] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8728.080861] Call Trace: [ 8728.080865] [ 8728.080870] ? show_regs.cold.9+0x22/0x2f [ 8728.080876] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8728.080881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8728.080883] ? watchdog+0x40/0x40 [ 8728.080886] ? hrtimer_interrupt+0x128/0x2c0 [ 8728.080889] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8728.080895] ? apic_timer_interrupt+0xf/0x20 [ 8728.080898] [ 8728.080900] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8728.080905] do_raw_spin_lock+0x177/0x190 [ 8728.080909] _raw_spin_lock+0x21/0x30 [ 8728.080912] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8728.085747] Modules linked in: [ 8728.090154] ? do_raw_spin_unlock+0x75/0x190 [ 8728.090176] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8728.090486] ? do_wait_intr+0xf0/0xf0 [ 8728.090495] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8728.090682] mdc_close+0x53b/0xd70 [mdc] [ 8728.090721] lmv_close+0x18d/0x490 [lmv] [ 8728.090751] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8728.091012] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8728.091090] ll_md_close+0x274/0x800 [lustre] [ 8728.091171] ll_file_release+0x128/0xc50 [lustre] [ 8728.091317] __fput+0x10d/0x440 [ 8728.091327] ____fput+0x12/0x20 [ 8728.091332] task_work_run+0xc6/0x110 [ 8728.091339] exit_to_usermode_loop+0x1dd/0x1f0 [ 8728.091345] do_syscall_64+0x3d6/0x440 [ 8728.091350] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8728.091359] RIP: 0033:0x7f1a63983cd8 [ 8728.091367] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8728.091371] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8728.091376] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8728.091379] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8728.091381] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8728.091383] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8728.091385] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8728.226179] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8728.245909] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8728.251312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8728.255831] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8728.258650] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8728.269912] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8728.273517] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8728.276597] RDX: 0000000000002dac RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8728.280498] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8728.283683] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8728.287577] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8728.290805] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8728.294037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8728.296656] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8728.299859] Call Trace: [ 8728.301037] [ 8728.301930] ? show_regs.cold.9+0x22/0x2f [ 8728.303942] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8728.306004] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8728.308087] ? watchdog+0x40/0x40 [ 8728.309329] ? hrtimer_interrupt+0x128/0x2c0 [ 8728.311334] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8728.313489] ? apic_timer_interrupt+0xf/0x20 [ 8728.315444] [ 8728.316435] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8728.318976] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8728.321528] do_raw_spin_lock+0x177/0x190 [ 8728.323239] _raw_spin_lock+0x21/0x30 [ 8728.324822] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8728.327252] obd_mod_free+0x46/0x140 [ptlrpc] [ 8728.329876] mdc_commit_open+0x91/0xb0 [mdc] [ 8728.331681] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8728.333562] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8728.336315] after_reply+0xe2c/0x1780 [ptlrpc] [ 8728.338648] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8728.341157] ? __next_timer_interrupt+0x160/0x160 [ 8728.343242] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8728.345467] ? do_wait_intr+0xf0/0xf0 [ 8728.347108] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8728.349600] kthread+0x1d1/0x200 [ 8728.350968] ? set_kthread_struct+0x70/0x70 [ 8728.352706] ret_from_fork+0x1f/0x30 [ 8732.058128] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8732.061035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8732.105017] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8732.111509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8732.118784] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8732.128521] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8732.144343] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8732.151698] RAX: 0000000000007f86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8732.158862] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8732.163177] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8732.168112] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8732.172787] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8732.180647] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8732.185717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8732.189056] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8732.194346] Call Trace: [ 8732.196298] [ 8732.197213] ? show_regs.cold.9+0x22/0x2f [ 8732.199973] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8732.202946] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8732.205697] ? watchdog+0x40/0x40 [ 8732.207655] ? hrtimer_interrupt+0x128/0x2c0 [ 8732.209778] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8732.213331] ? apic_timer_interrupt+0xf/0x20 [ 8732.215742] [ 8732.217135] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8732.219852] do_raw_spin_lock+0x177/0x190 [ 8732.222745] _raw_spin_lock+0x21/0x30 [ 8732.224360] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8732.227578] ? do_raw_spin_unlock+0x75/0x190 [ 8732.230351] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8732.233779] ? do_wait_intr+0xf0/0xf0 [ 8732.235761] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8732.239106] mdc_close+0x53b/0xd70 [mdc] [ 8732.244109] lmv_close+0x18d/0x490 [lmv] [ 8732.246186] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8732.249665] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8732.253957] ll_md_close+0x274/0x800 [lustre] [ 8732.256195] ll_file_release+0x128/0xc50 [lustre] [ 8732.262508] __fput+0x10d/0x440 [ 8732.266034] ____fput+0x12/0x20 [ 8732.267869] task_work_run+0xc6/0x110 [ 8732.272654] exit_to_usermode_loop+0x1dd/0x1f0 [ 8732.279151] do_syscall_64+0x3d6/0x440 [ 8732.281201] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8732.282701] RIP: 0033:0x7f1a63983cd8 [ 8732.290753] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8732.303833] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8732.311573] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8732.316016] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8732.318930] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8732.322280] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8732.325972] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8740.054950] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8740.057161] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8740.085550] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8740.091088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8740.097684] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8740.102078] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8740.115187] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8740.120593] RAX: 0000000000007715 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8740.123840] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8740.127755] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8740.133531] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8740.135801] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8740.143364] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8740.145975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8740.153579] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8740.161448] Call Trace: [ 8740.162090] [ 8740.162882] ? show_regs.cold.9+0x22/0x2f [ 8740.165742] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8740.169497] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8740.174704] ? watchdog+0x40/0x40 [ 8740.176047] ? hrtimer_interrupt+0x128/0x2c0 [ 8740.177988] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8740.182757] ? apic_timer_interrupt+0xf/0x20 [ 8740.184778] [ 8740.185614] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8740.192698] do_raw_spin_lock+0x177/0x190 [ 8740.197101] _raw_spin_lock+0x21/0x30 [ 8740.198515] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8740.212288] ? do_raw_spin_unlock+0x75/0x190 [ 8740.215867] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8740.225366] ? do_wait_intr+0xf0/0xf0 [ 8740.230135] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8740.238837] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8740.245771] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8740.252699] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8740.260679] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8740.262612] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8740.272079] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8740.274360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8740.279082] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8740.282748] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8740.284451] lmv_intent_open+0x338/0x1150 [lmv] [ 8740.290251] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8740.293573] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8740.296797] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8740.299072] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8740.302628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8740.308597] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8740.312438] ? do_raw_spin_unlock+0x75/0x190 [ 8740.315068] ? _raw_spin_unlock+0x12/0x30 [ 8740.320590] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8740.324085] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8740.327587] ll_atomic_open+0x36a/0xd70 [lustre] [ 8740.330077] lookup_open+0x356/0xb40 [ 8740.339742] do_last+0x313/0xfc0 [ 8740.342689] ? nd_jump_root+0xe5/0x160 [ 8740.347231] ? path_init+0x437/0x520 [ 8740.349030] path_openat+0xf7/0x500 [ 8740.358285] ? do_raw_spin_unlock+0x75/0x190 [ 8740.363064] do_filp_open+0x99/0x140 [ 8740.365647] ? getname_flags+0x6e/0x330 [ 8740.371075] ? __check_object_size+0xff/0x256 [ 8740.375459] ? do_raw_spin_unlock+0x75/0x190 [ 8740.378548] ? _raw_spin_unlock+0x12/0x30 [ 8740.380200] do_sys_openat2+0x2b4/0x410 [ 8740.382119] do_sys_open+0x73/0xa0 [ 8740.384094] __x64_sys_openat+0x24/0x30 [ 8740.387841] do_syscall_64+0xc1/0x440 [ 8740.389621] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8740.391891] RIP: 0033:0x7f1a63983332 [ 8740.393807] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8740.405290] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8740.409386] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8740.418127] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8740.420134] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8740.432964] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8740.435984] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8742.585533] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 8742.596988] rcu: 3-...!: (4227569 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 8742.614197] (detected by 1, t=6720179 jiffies, g=269633, q=2405) [ 8742.619888] Sending NMI from CPU 1 to CPUs 3: [ 8742.631451] NMI backtrace for cpu 3 [ 8742.631458] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8742.631461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8742.631462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8742.631464] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8742.631466] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 8742.631469] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8742.631470] RDX: 000000000000010e RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8742.631471] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8742.631473] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8742.631474] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8742.631475] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8742.631477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8742.631478] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8742.631479] Call Trace: [ 8742.631480] [ 8742.631481] ? show_regs.cold.9+0x22/0x2f [ 8742.631482] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8742.631483] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8742.631485] ? nmi_handle+0xbd/0x210 [ 8742.631486] ? default_do_nmi+0x4b/0x1b0 [ 8742.631487] ? do_nmi+0x280/0x380 [ 8742.631488] ? end_repeat_nmi+0x16/0x1f [ 8742.631489] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8742.631490] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8742.631491] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8742.631492] [ 8742.631493] do_raw_spin_lock+0x177/0x190 [ 8742.631494] _raw_spin_lock+0x21/0x30 [ 8742.631495] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8742.631496] obd_mod_free+0x46/0x140 [ptlrpc] [ 8742.631497] mdc_commit_open+0x91/0xb0 [mdc] [ 8742.631498] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8742.631639] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8742.631641] after_reply+0xe2c/0x1780 [ptlrpc] [ 8742.631642] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8742.631643] ? __next_timer_interrupt+0x160/0x160 [ 8742.631644] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8742.631645] ? do_wait_intr+0xf0/0xf0 [ 8742.631646] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8742.631647] kthread+0x1d1/0x200 [ 8742.631648] ? set_kthread_struct+0x70/0x70 [ 8742.631649] ret_from_fork+0x1f/0x30 [ 8742.637335] rcu: rcu_sched kthread starved for 6719825 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 8742.919945] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8742.933027] rcu: RCU grace-period kthread stack dump: [ 8742.935089] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8742.940794] Call Trace: [ 8742.942472] __schedule+0x351/0xcb0 [ 8742.943853] schedule+0xc0/0x180 [ 8742.947346] schedule_timeout+0xb4/0x190 [ 8742.950229] ? do_raw_spin_unlock+0x75/0x190 [ 8742.955216] ? __next_timer_interrupt+0x160/0x160 [ 8742.957041] ? prepare_to_swait_event+0xf3/0x140 [ 8742.960348] rcu_gp_kthread+0x8dd/0x15b0 [ 8742.962287] ? force_qs_rnp+0x2c0/0x2c0 [ 8742.964280] kthread+0x1d1/0x200 [ 8742.965514] ? set_kthread_struct+0x70/0x70 [ 8742.973943] ret_from_fork+0x1f/0x30 [ 8742.980588] rcu: Stack dump where RCU GP kthread last ran: [ 8742.988006] Sending NMI from CPU 1 to CPUs 3: [ 8742.995384] NMI backtrace for cpu 3 [ 8742.995391] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8742.995393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8742.995395] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8742.995396] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8742.995398] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 8742.995401] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8742.995403] RDX: 0000000000007bc9 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8742.995404] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8742.995406] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8742.995407] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8742.995409] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8742.995410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8742.995412] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8742.995413] Call Trace: [ 8742.995414] [ 8742.995415] ? show_regs.cold.9+0x22/0x2f [ 8742.995416] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8742.995418] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8742.995419] ? nmi_handle+0xbd/0x210 [ 8742.995420] ? default_do_nmi+0x4b/0x1b0 [ 8742.995421] ? do_nmi+0x280/0x380 [ 8742.995422] ? end_repeat_nmi+0x16/0x1f [ 8742.995424] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8742.995425] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8742.995427] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8742.995428] [ 8742.995429] do_raw_spin_lock+0x177/0x190 [ 8742.995430] _raw_spin_lock+0x21/0x30 [ 8742.995431] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8742.995432] obd_mod_free+0x46/0x140 [ptlrpc] [ 8742.995433] mdc_commit_open+0x91/0xb0 [mdc] [ 8742.995435] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8742.995436] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8742.995438] after_reply+0xe2c/0x1780 [ptlrpc] [ 8742.995439] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8742.995441] ? __next_timer_interrupt+0x160/0x160 [ 8742.995442] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8742.995443] ? do_wait_intr+0xf0/0xf0 [ 8742.995444] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8742.995445] kthread+0x1d1/0x200 [ 8742.995446] ? set_kthread_struct+0x70/0x70 [ 8742.995447] ret_from_fork+0x1f/0x30 [ 8743.001437] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6720s! [ 8743.192949] Showing busy workqueues and worker pools: [ 8743.197742] workqueue events: flags=0x0 [ 8743.201356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8743.206486] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8743.211695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8743.215936] pending: sec_gc_main [ptlrpc] [ 8743.220563] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8743.224742] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8743.232523] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8743.238002] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8743.241890] workqueue events_long: flags=0x0 [ 8743.243453] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8743.245665] pending: lu_objects_flush [obdclass] [ 8743.249183] workqueue events_unbound: flags=0x2 [ 8743.252545] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8743.256878] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8743.265116] workqueue events_power_efficient: flags=0x80 [ 8743.266911] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8743.272681] pending: neigh_periodic_work, do_cache_clean [ 8743.277392] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8743.280510] pending: neigh_periodic_work, check_lifetime [ 8743.285563] workqueue mm_percpu_wq: flags=0x8 [ 8743.287693] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8743.293682] pending: vmstat_update [ 8743.297636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8743.301454] pending: vmstat_update [ 8743.305507] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8743.308819] pending: vmstat_update [ 8743.312474] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8743.316613] pending: vmstat_update [ 8743.318993] workqueue writeback: flags=0x4a [ 8743.321717] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8743.326658] pending: wb_workfn, wb_workfn [ 8743.329586] workqueue xprtiod: flags=0xa [ 8743.331845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8743.336917] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8743.341725] workqueue ipv6_addrconf: flags=0x40008 [ 8743.343623] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8743.348966] pending: addrconf_verify_work [ 8743.352638] workqueue knbd0-recv: flags=0x1a [ 8743.355271] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8743.359721] in-flight: 647:recv_work [ 8743.364209] workqueue ptlrpc_pinger: flags=0x2 [ 8743.366448] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8743.368665] pending: ptlrpc_pinger_main [ptlrpc] [ 8743.372837] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8733s workers=2 idle: 652 [ 8760.057099] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8760.062031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8760.088124] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8760.096590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8760.102867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8760.106259] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8760.115799] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8760.119653] RAX: 0000000000007d8f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8760.121944] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8760.125283] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8760.128407] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8760.131832] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8760.137778] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8760.141473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8760.146225] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8760.149380] Call Trace: [ 8760.155404] [ 8760.158261] ? show_regs.cold.9+0x22/0x2f [ 8760.163601] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8760.167020] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8760.169150] ? watchdog+0x40/0x40 [ 8760.170597] ? hrtimer_interrupt+0x128/0x2c0 [ 8760.172510] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8760.174318] ? apic_timer_interrupt+0xf/0x20 [ 8760.176100] [ 8760.177205] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8760.179698] do_raw_spin_lock+0x177/0x190 [ 8760.182598] _raw_spin_lock+0x21/0x30 [ 8760.184270] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8760.187024] ? do_raw_spin_unlock+0x75/0x190 [ 8760.189180] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8760.191887] ? do_wait_intr+0xf0/0xf0 [ 8760.193618] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8760.196234] mdc_close+0x53b/0xd70 [mdc] [ 8760.197537] lmv_close+0x18d/0x490 [lmv] [ 8760.198752] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8760.201836] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8760.204093] ll_md_close+0x274/0x800 [lustre] [ 8760.206219] ll_file_release+0x128/0xc50 [lustre] [ 8760.208287] __fput+0x10d/0x440 [ 8760.209331] ____fput+0x12/0x20 [ 8760.212783] task_work_run+0xc6/0x110 [ 8760.215805] exit_to_usermode_loop+0x1dd/0x1f0 [ 8760.221694] do_syscall_64+0x3d6/0x440 [ 8760.223915] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8760.230607] RIP: 0033:0x7f1a63983cd8 [ 8760.232595] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8760.248782] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8760.252502] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8760.255903] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8760.259068] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8760.262193] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8760.265353] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8768.053021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8768.055628] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8768.056695] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8768.058016] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8768.058024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8768.058099] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8768.058105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8768.058119] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8768.058134] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8768.058138] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8768.058142] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8768.058144] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8768.058146] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8768.058148] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8768.058150] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8768.058153] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8768.058155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8768.058157] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8768.058165] Call Trace: [ 8768.058171] [ 8768.058178] ? show_regs.cold.9+0x22/0x2f [ 8768.058190] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8768.058196] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8768.058201] ? watchdog+0x40/0x40 [ 8768.058205] ? hrtimer_interrupt+0x128/0x2c0 [ 8768.058210] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8768.058217] ? apic_timer_interrupt+0xf/0x20 [ 8768.058221] [ 8768.058225] ? native_safe_halt+0x2e/0x50 [ 8768.058229] kvm_wait+0xcd/0xe0 [ 8768.058236] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8768.058244] do_raw_spin_lock+0x177/0x190 [ 8768.058248] _raw_spin_lock+0x21/0x30 [ 8768.058254] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8768.058523] obd_mod_free+0x46/0x140 [ptlrpc] [ 8768.058713] mdc_commit_open+0x91/0xb0 [mdc] [ 8768.058762] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8768.058975] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8768.059175] after_reply+0xe2c/0x1780 [ptlrpc] [ 8768.059370] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8768.059561] ? __next_timer_interrupt+0x160/0x160 [ 8768.059569] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8768.059768] ? do_wait_intr+0xf0/0xf0 [ 8768.059778] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8768.059989] kthread+0x1d1/0x200 [ 8768.059994] ? set_kthread_struct+0x70/0x70 [ 8768.059998] ret_from_fork+0x1f/0x30 [ 8768.110294] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8768.113339] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8768.113423] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8768.113430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8768.113434] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8768.113449] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8768.113453] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8768.113457] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8768.113460] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8768.113462] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8768.113464] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8768.113466] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8768.113469] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8768.113472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8768.113475] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8768.113484] Call Trace: [ 8768.113490] [ 8768.113496] ? show_regs.cold.9+0x22/0x2f [ 8768.113507] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8768.113514] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8768.113519] ? watchdog+0x40/0x40 [ 8768.113523] ? hrtimer_interrupt+0x128/0x2c0 [ 8768.113527] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8768.113535] ? apic_timer_interrupt+0xf/0x20 [ 8768.113539] [ 8768.113542] ? native_safe_halt+0x2e/0x50 [ 8768.113546] kvm_wait+0xcd/0xe0 [ 8768.113554] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8768.113563] do_raw_spin_lock+0x177/0x190 [ 8768.113567] _raw_spin_lock+0x21/0x30 [ 8768.113571] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8768.121603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8768.139938] ? do_raw_spin_unlock+0x75/0x190 [ 8768.140012] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8768.140304] ? do_wait_intr+0xf0/0xf0 [ 8768.140312] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8768.140502] mdc_close+0x53b/0xd70 [mdc] [ 8768.140543] lmv_close+0x18d/0x490 [lmv] [ 8768.140574] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8768.140683] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8768.140762] ll_md_close+0x274/0x800 [lustre] [ 8768.140845] ll_file_release+0x128/0xc50 [lustre] [ 8768.141151] __fput+0x10d/0x440 [ 8768.141168] ____fput+0x12/0x20 [ 8768.141173] task_work_run+0xc6/0x110 [ 8768.141181] exit_to_usermode_loop+0x1dd/0x1f0 [ 8768.141188] do_syscall_64+0x3d6/0x440 [ 8768.141193] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8768.141201] RIP: 0033:0x7f1a63983cd8 [ 8768.141210] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8768.141214] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8768.141220] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8768.141228] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8768.141230] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8768.141233] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8768.141236] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8768.531085] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8768.532335] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8768.539652] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8768.543505] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8768.547290] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8768.551205] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8768.554418] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8768.561109] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8768.565071] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8768.569382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8768.571762] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8768.576062] Call Trace: [ 8768.577508] [ 8768.578773] ? show_regs.cold.9+0x22/0x2f [ 8768.580473] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8768.582631] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8768.584588] ? watchdog+0x40/0x40 [ 8768.589269] ? hrtimer_interrupt+0x128/0x2c0 [ 8768.591444] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8768.594238] ? apic_timer_interrupt+0xf/0x20 [ 8768.596831] [ 8768.597738] ? native_safe_halt+0x2e/0x50 [ 8768.600117] kvm_wait+0xcd/0xe0 [ 8768.601631] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8768.604518] do_raw_spin_lock+0x177/0x190 [ 8768.606022] _raw_spin_lock+0x21/0x30 [ 8768.608263] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8768.610985] ? do_raw_spin_unlock+0x75/0x190 [ 8768.613309] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8768.617010] ? do_wait_intr+0xf0/0xf0 [ 8768.619574] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8768.622372] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8768.625289] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8768.627274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8768.629401] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8768.632015] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8768.634179] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8768.636297] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8768.639015] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8768.642131] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8768.644629] lmv_intent_open+0x338/0x1150 [lmv] [ 8768.646859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8768.649404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8768.652083] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8768.654182] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8768.656449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8768.659102] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8768.661712] ? do_raw_spin_unlock+0x75/0x190 [ 8768.664038] ? _raw_spin_unlock+0x12/0x30 [ 8768.666023] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8768.668713] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8768.671372] ll_atomic_open+0x36a/0xd70 [lustre] [ 8768.673802] lookup_open+0x356/0xb40 [ 8768.675600] do_last+0x313/0xfc0 [ 8768.677350] ? nd_jump_root+0xe5/0x160 [ 8768.679293] ? path_init+0x437/0x520 [ 8768.680697] path_openat+0xf7/0x500 [ 8768.682264] ? do_raw_spin_unlock+0x75/0x190 [ 8768.683981] do_filp_open+0x99/0x140 [ 8768.685941] ? getname_flags+0x6e/0x330 [ 8768.687801] ? __check_object_size+0xff/0x256 [ 8768.690365] ? do_raw_spin_unlock+0x75/0x190 [ 8768.692608] ? _raw_spin_unlock+0x12/0x30 [ 8768.694795] do_sys_openat2+0x2b4/0x410 [ 8768.696586] do_sys_open+0x73/0xa0 [ 8768.697808] __x64_sys_openat+0x24/0x30 [ 8768.699625] do_syscall_64+0xc1/0x440 [ 8768.701565] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8768.704004] RIP: 0033:0x7f1a63983332 [ 8768.706011] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8768.719359] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8768.725827] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8768.736378] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8768.741777] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8768.752023] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8768.761563] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8773.600082] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6750s! [ 8773.604590] Showing busy workqueues and worker pools: [ 8773.607487] workqueue events: flags=0x0 [ 8773.609647] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8773.617879] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8773.624182] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8773.629221] pending: sec_gc_main [ptlrpc] [ 8773.632193] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8773.636556] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8773.643011] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8773.647554] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8773.653467] workqueue events_long: flags=0x0 [ 8773.656665] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8773.662200] pending: lu_objects_flush [obdclass] [ 8773.666708] workqueue events_unbound: flags=0x2 [ 8773.672457] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8773.680456] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8773.695197] workqueue events_power_efficient: flags=0x80 [ 8773.698908] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8773.705883] pending: neigh_periodic_work, do_cache_clean [ 8773.709488] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8773.716218] pending: neigh_periodic_work, check_lifetime [ 8773.724386] workqueue mm_percpu_wq: flags=0x8 [ 8773.730069] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8773.732547] pending: vmstat_update [ 8773.744780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8773.757667] pending: vmstat_update [ 8773.765362] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8773.771392] pending: vmstat_update [ 8773.777534] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8773.782101] pending: vmstat_update [ 8773.784380] workqueue writeback: flags=0x4a [ 8773.787291] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8773.795341] pending: wb_workfn, wb_workfn [ 8773.803292] workqueue xprtiod: flags=0xa [ 8773.805399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8773.809838] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8773.817418] workqueue ipv6_addrconf: flags=0x40008 [ 8773.821483] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8773.833150] pending: addrconf_verify_work [ 8773.836564] workqueue knbd0-recv: flags=0x1a [ 8773.844465] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8773.854431] in-flight: 647:recv_work [ 8773.858139] workqueue ptlrpc_pinger: flags=0x2 [ 8773.865648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8773.878901] pending: ptlrpc_pinger_main [ptlrpc] [ 8773.884133] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8763s workers=2 idle: 652 [ 8788.058750] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8788.065582] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8788.146820] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8788.170961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8788.183508] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8788.193225] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8788.212137] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8788.232881] RAX: 000000000000727a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8788.253222] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8788.265017] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8788.276821] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8788.300959] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8788.303062] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8788.312424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8788.317238] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8788.321955] Call Trace: [ 8788.323257] [ 8788.324412] ? show_regs.cold.9+0x22/0x2f [ 8788.326265] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8788.331396] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8788.339620] ? watchdog+0x40/0x40 [ 8788.342483] ? hrtimer_interrupt+0x128/0x2c0 [ 8788.344409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8788.345981] ? apic_timer_interrupt+0xf/0x20 [ 8788.347014] [ 8788.347488] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8788.351717] do_raw_spin_lock+0x177/0x190 [ 8788.354278] _raw_spin_lock+0x21/0x30 [ 8788.355492] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8788.357380] ? do_raw_spin_unlock+0x75/0x190 [ 8788.358444] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8788.369019] ? do_wait_intr+0xf0/0xf0 [ 8788.373982] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8788.379821] mdc_close+0x53b/0xd70 [mdc] [ 8788.384059] lmv_close+0x18d/0x490 [lmv] [ 8788.389388] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8788.393891] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8788.397999] ll_md_close+0x274/0x800 [lustre] [ 8788.403285] ll_file_release+0x128/0xc50 [lustre] [ 8788.407940] __fput+0x10d/0x440 [ 8788.410455] ____fput+0x12/0x20 [ 8788.411971] task_work_run+0xc6/0x110 [ 8788.415224] exit_to_usermode_loop+0x1dd/0x1f0 [ 8788.423315] do_syscall_64+0x3d6/0x440 [ 8788.431038] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8788.435946] RIP: 0033:0x7f1a63983cd8 [ 8788.440141] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8788.456094] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8788.461075] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8788.474256] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8788.476232] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8788.481502] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8788.491303] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8796.058014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8796.058248] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8796.061214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8796.061309] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8796.061319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8796.061322] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8796.061337] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8796.061579] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8796.061589] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8796.061592] RDX: 0000000000006eff RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8796.061594] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8796.061615] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8796.061618] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8796.061622] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8796.061625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8796.061628] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8796.061635] Call Trace: [ 8796.061641] [ 8796.061646] ? show_regs.cold.9+0x22/0x2f [ 8796.061661] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8796.061671] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8796.061678] ? watchdog+0x40/0x40 [ 8796.061682] ? hrtimer_interrupt+0x128/0x2c0 [ 8796.061687] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8796.061695] ? apic_timer_interrupt+0xf/0x20 [ 8796.061700] [ 8796.061704] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8796.061709] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8796.061717] do_raw_spin_lock+0x177/0x190 [ 8796.061721] _raw_spin_lock+0x21/0x30 [ 8796.061727] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8796.062285] obd_mod_free+0x46/0x140 [ptlrpc] [ 8796.062493] mdc_commit_open+0x91/0xb0 [mdc] [ 8796.062532] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8796.069573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [ 8796.093803] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8796.104503] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 8796.109555] after_reply+0xe2c/0x1780 [ptlrpc] [ 8796.110089] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8796.110306] ? __next_timer_interrupt+0x160/0x160 [ 8796.110317] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8796.110591] ? do_wait_intr+0xf0/0xf0 [ 8796.110602] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8796.110933] kthread+0x1d1/0x200 [ 8796.110940] ? set_kthread_struct+0x70/0x70 [ 8796.111004] ret_from_fork+0x1f/0x30 [ 8796.226134] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8796.250713] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8796.257463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8796.260453] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8796.263742] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8796.272590] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8796.276227] RAX: 0000000000002ec2 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8796.279442] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8796.282421] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8796.285451] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8796.288423] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8796.291061] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8796.293972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8796.296147] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8796.301287] Call Trace: [ 8796.302276] [ 8796.302928] ? show_regs.cold.9+0x22/0x2f [ 8796.304208] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8796.306117] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8796.307344] ? watchdog+0x40/0x40 [ 8796.308781] ? hrtimer_interrupt+0x128/0x2c0 [ 8796.310096] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8796.314108] ? apic_timer_interrupt+0xf/0x20 [ 8796.319220] [ 8796.321022] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8796.324982] do_raw_spin_lock+0x177/0x190 [ 8796.329455] _raw_spin_lock+0x21/0x30 [ 8796.332289] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8796.336238] ? do_raw_spin_unlock+0x75/0x190 [ 8796.337748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8796.339872] ? do_wait_intr+0xf0/0xf0 [ 8796.341088] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8796.343180] mdc_close+0x53b/0xd70 [mdc] [ 8796.344884] lmv_close+0x18d/0x490 [lmv] [ 8796.346255] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8796.349282] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8796.351443] ll_md_close+0x274/0x800 [lustre] [ 8796.353528] ll_file_release+0x128/0xc50 [lustre] [ 8796.356004] __fput+0x10d/0x440 [ 8796.357206] ____fput+0x12/0x20 [ 8796.358565] task_work_run+0xc6/0x110 [ 8796.360214] exit_to_usermode_loop+0x1dd/0x1f0 [ 8796.362188] do_syscall_64+0x3d6/0x440 [ 8796.363873] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8796.366323] RIP: 0033:0x7f1a63983cd8 [ 8796.368093] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8796.381721] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8796.385240] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8796.391660] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8796.396179] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8796.399312] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8796.402507] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8800.053012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8800.061715] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8800.105292] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8800.113855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8800.122964] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8800.127791] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8800.145656] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8800.151212] RAX: 0000000000001610 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8800.155396] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8800.158949] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8800.164591] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8800.172063] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8800.178473] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8800.185032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8800.189113] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8800.195235] Call Trace: [ 8800.196843] [ 8800.198894] ? show_regs.cold.9+0x22/0x2f [ 8800.206131] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8800.212219] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8800.219835] ? watchdog+0x40/0x40 [ 8800.222195] ? hrtimer_interrupt+0x128/0x2c0 [ 8800.229079] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8800.232965] ? apic_timer_interrupt+0xf/0x20 [ 8800.235978] [ 8800.238107] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8800.244137] do_raw_spin_lock+0x177/0x190 [ 8800.247756] _raw_spin_lock+0x21/0x30 [ 8800.249161] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8800.253980] ? do_raw_spin_unlock+0x75/0x190 [ 8800.258048] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8800.263321] ? do_wait_intr+0xf0/0xf0 [ 8800.269941] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8800.276425] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8800.282866] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8800.287975] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8800.291473] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8800.294468] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8800.296928] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8800.298978] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8800.301459] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8800.304986] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8800.307522] lmv_intent_open+0x338/0x1150 [lmv] [ 8800.309785] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8800.312218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8800.314928] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8800.316975] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8800.319301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8800.322440] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8800.324362] ? do_raw_spin_unlock+0x75/0x190 [ 8800.326890] ? _raw_spin_unlock+0x12/0x30 [ 8800.328151] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8800.330543] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8800.334072] ll_atomic_open+0x36a/0xd70 [lustre] [ 8800.336375] lookup_open+0x356/0xb40 [ 8800.337412] do_last+0x313/0xfc0 [ 8800.343376] ? nd_jump_root+0xe5/0x160 [ 8800.345135] ? path_init+0x437/0x520 [ 8800.349942] path_openat+0xf7/0x500 [ 8800.353239] ? do_raw_spin_unlock+0x75/0x190 [ 8800.356134] do_filp_open+0x99/0x140 [ 8800.363227] ? getname_flags+0x6e/0x330 [ 8800.368718] ? __check_object_size+0xff/0x256 [ 8800.374900] ? do_raw_spin_unlock+0x75/0x190 [ 8800.378993] ? _raw_spin_unlock+0x12/0x30 [ 8800.380515] do_sys_openat2+0x2b4/0x410 [ 8800.385034] do_sys_open+0x73/0xa0 [ 8800.386359] __x64_sys_openat+0x24/0x30 [ 8800.390931] do_syscall_64+0xc1/0x440 [ 8800.392847] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8800.395247] RIP: 0033:0x7f1a63983332 [ 8800.396814] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8800.404521] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8800.410760] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8800.422466] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8800.428996] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8800.441256] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8800.445979] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8804.321386] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6781s! [ 8804.327576] Showing busy workqueues and worker pools: [ 8804.332022] workqueue events: flags=0x0 [ 8804.333295] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8804.342738] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8804.352155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8804.361509] pending: sec_gc_main [ptlrpc] [ 8804.369325] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8804.379747] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8804.394336] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8804.398611] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8804.409995] workqueue events_long: flags=0x0 [ 8804.412681] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8804.420308] pending: lu_objects_flush [obdclass] [ 8804.430369] workqueue events_unbound: flags=0x2 [ 8804.433465] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8804.439203] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8804.444546] workqueue events_power_efficient: flags=0x80 [ 8804.446304] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8804.459689] pending: neigh_periodic_work, do_cache_clean [ 8804.462534] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8804.469357] pending: neigh_periodic_work, check_lifetime [ 8804.474632] workqueue mm_percpu_wq: flags=0x8 [ 8804.479030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8804.485180] pending: vmstat_update [ 8804.487373] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8804.508359] pending: vmstat_update [ 8804.512457] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8804.518345] pending: vmstat_update [ 8804.520683] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8804.524242] pending: vmstat_update [ 8804.526215] workqueue writeback: flags=0x4a [ 8804.528408] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8804.531746] pending: wb_workfn, wb_workfn [ 8804.534318] workqueue xprtiod: flags=0xa [ 8804.536399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8804.544564] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8804.558018] workqueue ipv6_addrconf: flags=0x40008 [ 8804.561979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8804.569135] pending: addrconf_verify_work [ 8804.576458] workqueue knbd0-recv: flags=0x1a [ 8804.580234] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8804.586478] in-flight: 647:recv_work [ 8804.592120] workqueue ptlrpc_pinger: flags=0x2 [ 8804.598574] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8804.603162] pending: ptlrpc_pinger_main [ptlrpc] [ 8804.609072] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8794s workers=2 idle: 652 [ 8816.060990] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8816.064394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8816.115105] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8816.126827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8816.134739] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8816.140745] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8816.168487] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8816.179727] RAX: 0000000000002796 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8816.184916] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8816.191370] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8816.199029] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8816.204578] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8816.212385] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8816.218344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8816.221358] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8816.226075] Call Trace: [ 8816.229397] [ 8816.230475] ? show_regs.cold.9+0x22/0x2f [ 8816.232476] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8816.234743] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8816.238753] ? watchdog+0x40/0x40 [ 8816.242341] ? hrtimer_interrupt+0x128/0x2c0 [ 8816.244911] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8816.250508] ? apic_timer_interrupt+0xf/0x20 [ 8816.256206] [ 8816.258381] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8816.261999] do_raw_spin_lock+0x177/0x190 [ 8816.264356] _raw_spin_lock+0x21/0x30 [ 8816.267096] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8816.270348] ? do_raw_spin_unlock+0x75/0x190 [ 8816.273104] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8816.276991] ? do_wait_intr+0xf0/0xf0 [ 8816.279073] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8816.282377] mdc_close+0x53b/0xd70 [mdc] [ 8816.284738] lmv_close+0x18d/0x490 [lmv] [ 8816.288371] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8816.291726] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8816.294508] ll_md_close+0x274/0x800 [lustre] [ 8816.296981] ll_file_release+0x128/0xc50 [lustre] [ 8816.299703] __fput+0x10d/0x440 [ 8816.305042] ____fput+0x12/0x20 [ 8816.306506] task_work_run+0xc6/0x110 [ 8816.308458] exit_to_usermode_loop+0x1dd/0x1f0 [ 8816.310558] do_syscall_64+0x3d6/0x440 [ 8816.315996] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8816.319198] RIP: 0033:0x7f1a63983cd8 [ 8816.321235] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8816.337001] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8816.341497] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8816.352311] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8816.358460] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8816.362983] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8816.368300] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8824.056018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 8824.058038] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8824.060154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8824.060237] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8824.060244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8824.060247] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8824.060260] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8824.060265] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8824.060269] RAX: 0000000000002262 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8824.060271] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8824.060274] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8824.060275] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8824.060278] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8824.060282] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8824.060285] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8824.060287] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8824.060297] Call Trace: [ 8824.060302] [ 8824.060308] ? show_regs.cold.9+0x22/0x2f [ 8824.060317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8824.060323] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8824.060400] ? watchdog+0x40/0x40 [ 8824.060405] ? hrtimer_interrupt+0x128/0x2c0 [ 8824.060409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8824.060418] ? apic_timer_interrupt+0xf/0x20 [ 8824.060422] [ 8824.060425] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8824.060431] do_raw_spin_lock+0x177/0x190 [ 8824.060438] _raw_spin_lock+0x21/0x30 [ 8824.060443] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8824.065906] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8824.065996] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8824.066003] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8824.066006] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8824.066021] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8824.066025] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8824.066037] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8824.066039] RDX: 000000000000138d RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8824.066041] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8824.066044] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8824.066047] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8824.066050] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8824.066053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8824.066055] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8824.066063] Call Trace: [ 8824.066068] [ 8824.066074] ? show_regs.cold.9+0x22/0x2f [ 8824.066084] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8824.066090] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8824.066095] ? watchdog+0x40/0x40 [ 8824.066099] ? hrtimer_interrupt+0x128/0x2c0 [ 8824.066104] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8824.066112] ? apic_timer_interrupt+0xf/0x20 [ 8824.066117] [ 8824.066120] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8824.066125] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8824.066130] do_raw_spin_lock+0x177/0x190 [ 8824.066141] _raw_spin_lock+0x21/0x30 [ 8824.066147] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8824.092363] ? do_raw_spin_unlock+0x75/0x190 [ 8824.092391] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8824.092879] ? do_wait_intr+0xf0/0xf0 [ 8824.092887] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8824.093103] mdc_close+0x53b/0xd70 [mdc] [ 8824.093153] lmv_close+0x18d/0x490 [lmv] [ 8824.093182] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8824.093305] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8824.093469] ll_md_close+0x274/0x800 [lustre] [ 8824.093561] ll_file_release+0x128/0xc50 [lustre] [ 8824.093644] __fput+0x10d/0x440 [ 8824.093653] ____fput+0x12/0x20 [ 8824.093657] task_work_run+0xc6/0x110 [ 8824.093665] exit_to_usermode_loop+0x1dd/0x1f0 [ 8824.093677] do_syscall_64+0x3d6/0x440 [ 8824.093682] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8824.093690] RIP: 0033:0x7f1a63983cd8 [ 8824.093698] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8824.093702] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8824.093707] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8824.093710] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8824.093712] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8824.093715] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8824.093717] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8824.311469] obd_mod_free+0x46/0x140 [ptlrpc] [ 8824.313693] mdc_commit_open+0x91/0xb0 [mdc] [ 8824.314958] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8824.317480] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8824.319809] after_reply+0xe2c/0x1780 [ptlrpc] [ 8824.321871] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8824.323768] ? __next_timer_interrupt+0x160/0x160 [ 8824.325582] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8824.327116] ? do_wait_intr+0xf0/0xf0 [ 8824.328172] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8824.330609] kthread+0x1d1/0x200 [ 8824.331961] ? set_kthread_struct+0x70/0x70 [ 8824.333610] ret_from_fork+0x1f/0x30 [ 8832.053027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8832.061714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8832.093960] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8832.099699] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8832.104973] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8832.110714] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8832.129633] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8832.136236] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8832.143076] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8832.148960] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8832.154148] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8832.157592] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8832.161980] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8832.165308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8832.168898] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8832.172101] Call Trace: [ 8832.173629] [ 8832.178023] ? show_regs.cold.9+0x22/0x2f [ 8832.181807] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8832.188599] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8832.194022] ? watchdog+0x40/0x40 [ 8832.195944] ? hrtimer_interrupt+0x128/0x2c0 [ 8832.198702] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8832.201772] ? apic_timer_interrupt+0xf/0x20 [ 8832.205253] [ 8832.206186] ? native_safe_halt+0x2e/0x50 [ 8832.211736] kvm_wait+0xcd/0xe0 [ 8832.213125] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8832.219086] do_raw_spin_lock+0x177/0x190 [ 8832.221139] _raw_spin_lock+0x21/0x30 [ 8832.225607] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8832.229727] ? do_raw_spin_unlock+0x75/0x190 [ 8832.232813] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8832.235190] ? do_wait_intr+0xf0/0xf0 [ 8832.237062] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8832.239905] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8832.243610] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8832.250090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8832.252443] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8832.254106] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8832.258644] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8832.261162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8832.263418] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8832.266058] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8832.268951] lmv_intent_open+0x338/0x1150 [lmv] [ 8832.271560] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8832.273048] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8832.275631] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8832.279126] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8832.282623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8832.285743] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8832.288543] ? do_raw_spin_unlock+0x75/0x190 [ 8832.290310] ? _raw_spin_unlock+0x12/0x30 [ 8832.292125] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8832.294783] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8832.297133] ll_atomic_open+0x36a/0xd70 [lustre] [ 8832.299388] lookup_open+0x356/0xb40 [ 8832.301341] do_last+0x313/0xfc0 [ 8832.302762] ? nd_jump_root+0xe5/0x160 [ 8832.304914] ? path_init+0x437/0x520 [ 8832.306631] path_openat+0xf7/0x500 [ 8832.308472] ? do_raw_spin_unlock+0x75/0x190 [ 8832.310624] do_filp_open+0x99/0x140 [ 8832.312516] ? getname_flags+0x6e/0x330 [ 8832.313986] ? __check_object_size+0xff/0x256 [ 8832.315923] ? do_raw_spin_unlock+0x75/0x190 [ 8832.317476] ? _raw_spin_unlock+0x12/0x30 [ 8832.319018] do_sys_openat2+0x2b4/0x410 [ 8832.320953] do_sys_open+0x73/0xa0 [ 8832.322812] __x64_sys_openat+0x24/0x30 [ 8832.324659] do_syscall_64+0xc1/0x440 [ 8832.326274] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8832.328572] RIP: 0033:0x7f1a63983332 [ 8832.329992] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8832.338651] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8832.341747] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8832.345198] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8832.348320] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8832.351298] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8832.354597] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8835.040138] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6812s! [ 8835.044031] Showing busy workqueues and worker pools: [ 8835.047397] workqueue events: flags=0x0 [ 8835.049591] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8835.053292] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8835.056594] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8835.060383] pending: sec_gc_main [ptlrpc] [ 8835.063348] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8835.066481] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8835.071474] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8835.074436] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8835.078581] workqueue events_long: flags=0x0 [ 8835.080021] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8835.082809] pending: lu_objects_flush [obdclass] [ 8835.085482] workqueue events_unbound: flags=0x2 [ 8835.087995] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8835.090963] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8835.095154] workqueue events_power_efficient: flags=0x80 [ 8835.097476] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8835.101047] pending: neigh_periodic_work, do_cache_clean [ 8835.103520] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8835.106617] pending: neigh_periodic_work, check_lifetime [ 8835.109579] workqueue mm_percpu_wq: flags=0x8 [ 8835.112019] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8835.115512] pending: vmstat_update [ 8835.117116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8835.120266] pending: vmstat_update [ 8835.122075] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8835.124480] pending: vmstat_update [ 8835.125858] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8835.128498] pending: vmstat_update [ 8835.130087] workqueue writeback: flags=0x4a [ 8835.131857] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8835.134394] pending: wb_workfn, wb_workfn [ 8835.136156] workqueue xprtiod: flags=0xa [ 8835.137899] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8835.140594] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8835.143612] workqueue ipv6_addrconf: flags=0x40008 [ 8835.145433] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8835.147958] pending: addrconf_verify_work [ 8835.149902] workqueue knbd0-recv: flags=0x1a [ 8835.151484] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8835.153574] in-flight: 647:recv_work [ 8835.155216] workqueue ptlrpc_pinger: flags=0x2 [ 8835.157635] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8835.160064] pending: ptlrpc_pinger_main [ptlrpc] [ 8835.163461] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8825s workers=2 idle: 652 [ 8844.060059] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8844.063843] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8844.097841] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8844.104880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8844.109966] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8844.113509] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8844.125362] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8844.129665] RAX: 0000000000007f83 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8844.135128] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8844.142477] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8844.147535] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8844.153668] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8844.157275] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8844.160277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8844.162839] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8844.169797] Call Trace: [ 8844.170774] [ 8844.171836] ? show_regs.cold.9+0x22/0x2f [ 8844.175656] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8844.182350] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8844.183894] ? watchdog+0x40/0x40 [ 8844.186256] ? hrtimer_interrupt+0x128/0x2c0 [ 8844.189588] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8844.191923] ? apic_timer_interrupt+0xf/0x20 [ 8844.193754] [ 8844.194664] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8844.196737] do_raw_spin_lock+0x177/0x190 [ 8844.198001] _raw_spin_lock+0x21/0x30 [ 8844.199487] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8844.201631] ? do_raw_spin_unlock+0x75/0x190 [ 8844.204335] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8844.206473] ? do_wait_intr+0xf0/0xf0 [ 8844.207984] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8844.210127] mdc_close+0x53b/0xd70 [mdc] [ 8844.211929] lmv_close+0x18d/0x490 [lmv] [ 8844.215389] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8844.217597] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8844.221131] ll_md_close+0x274/0x800 [lustre] [ 8844.226680] ll_file_release+0x128/0xc50 [lustre] [ 8844.233463] __fput+0x10d/0x440 [ 8844.235367] ____fput+0x12/0x20 [ 8844.237666] task_work_run+0xc6/0x110 [ 8844.239659] exit_to_usermode_loop+0x1dd/0x1f0 [ 8844.241477] do_syscall_64+0x3d6/0x440 [ 8844.244305] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8844.250263] RIP: 0033:0x7f1a63983cd8 [ 8844.251272] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8844.263282] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8844.271438] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8844.276387] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8844.288310] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8844.290985] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8844.294772] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8852.061506] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8852.061686] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 8852.070675] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8852.070744] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8852.070750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8852.070753] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8852.070765] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8852.070769] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8852.070773] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8852.070775] RDX: 0000000000003454 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8852.070777] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8852.070779] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8852.070781] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8852.070783] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8852.070786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8852.070788] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8852.070796] Call Trace: [ 8852.070801] [ 8852.070806] ? show_regs.cold.9+0x22/0x2f [ 8852.070814] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8852.070819] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8852.070822] ? watchdog+0x40/0x40 [ 8852.070826] ? hrtimer_interrupt+0x128/0x2c0 [ 8852.070829] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8852.070837] ? apic_timer_interrupt+0xf/0x20 [ 8852.070840] [ 8852.070843] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8852.070848] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8852.070852] do_raw_spin_lock+0x177/0x190 [ 8852.070857] _raw_spin_lock+0x21/0x30 [ 8852.070862] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8852.075898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8852.075982] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8852.075988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8852.075991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8852.076005] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8852.076009] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8852.076016] RAX: 0000000000007f89 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8852.076019] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8852.076021] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8852.076024] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8852.076026] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8852.076030] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8852.076033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8852.076035] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8852.076047] Call Trace: [ 8852.076052] [ 8852.076123] ? show_regs.cold.9+0x22/0x2f [ 8852.076134] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8852.076142] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8852.076146] ? watchdog+0x40/0x40 [ 8852.076151] ? hrtimer_interrupt+0x128/0x2c0 [ 8852.076155] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8852.076170] ? apic_timer_interrupt+0xf/0x20 [ 8852.076175] [ 8852.076178] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8852.076184] do_raw_spin_lock+0x177/0x190 [ 8852.076189] _raw_spin_lock+0x21/0x30 [ 8852.076194] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8852.077195] ? do_raw_spin_unlock+0x75/0x190 [ 8852.077204] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8852.077434] ? do_wait_intr+0xf0/0xf0 [ 8852.077441] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8852.077631] mdc_close+0x53b/0xd70 [mdc] [ 8852.077746] lmv_close+0x18d/0x490 [lmv] [ 8852.077778] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8852.078027] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8852.078122] ll_md_close+0x274/0x800 [lustre] [ 8852.078203] ll_file_release+0x128/0xc50 [lustre] [ 8852.078287] __fput+0x10d/0x440 [ 8852.078296] ____fput+0x12/0x20 [ 8852.078300] task_work_run+0xc6/0x110 [ 8852.078307] exit_to_usermode_loop+0x1dd/0x1f0 [ 8852.078314] do_syscall_64+0x3d6/0x440 [ 8852.078318] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8852.078327] RIP: 0033:0x7f1a63983cd8 [ 8852.078334] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8852.078338] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8852.078344] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8852.078346] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8852.078349] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8852.078351] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8852.078353] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8852.649578] obd_mod_free+0x46/0x140 [ptlrpc] [ 8852.655237] mdc_commit_open+0x91/0xb0 [mdc] [ 8852.658179] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8852.665113] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8852.669134] after_reply+0xe2c/0x1780 [ptlrpc] [ 8852.675574] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8852.684444] ? __next_timer_interrupt+0x160/0x160 [ 8852.690242] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8852.696271] ? do_wait_intr+0xf0/0xf0 [ 8852.702603] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8852.710642] kthread+0x1d1/0x200 [ 8852.714705] ? set_kthread_struct+0x70/0x70 [ 8852.721278] ret_from_fork+0x1f/0x30 [ 8860.055246] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8860.060514] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8860.108657] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8860.124017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8860.128354] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8860.132329] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8860.151126] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8860.153902] RAX: 0000000000007e88 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8860.166541] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8860.172253] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8860.177060] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8860.181098] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8860.186973] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8860.196550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8860.203945] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8860.211803] Call Trace: [ 8860.213543] [ 8860.214778] ? show_regs.cold.9+0x22/0x2f [ 8860.216464] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8860.218261] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8860.222086] ? watchdog+0x40/0x40 [ 8860.225877] ? hrtimer_interrupt+0x128/0x2c0 [ 8860.229908] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8860.233083] ? apic_timer_interrupt+0xf/0x20 [ 8860.239546] [ 8860.241789] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8860.246588] do_raw_spin_lock+0x177/0x190 [ 8860.248770] _raw_spin_lock+0x21/0x30 [ 8860.256654] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8860.264796] ? do_raw_spin_unlock+0x75/0x190 [ 8860.271957] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8860.275622] ? do_wait_intr+0xf0/0xf0 [ 8860.277877] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8860.281648] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8860.284605] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8860.288349] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8860.291843] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8860.294801] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8860.297537] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8860.299825] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8860.303858] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8860.307939] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8860.310838] lmv_intent_open+0x338/0x1150 [lmv] [ 8860.313994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8860.321772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8860.325451] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8860.333803] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8860.341471] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8860.345710] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8860.349882] ? do_raw_spin_unlock+0x75/0x190 [ 8860.351572] ? _raw_spin_unlock+0x12/0x30 [ 8860.355428] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8860.357330] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8860.360609] ll_atomic_open+0x36a/0xd70 [lustre] [ 8860.364342] lookup_open+0x356/0xb40 [ 8860.366788] do_last+0x313/0xfc0 [ 8860.372484] ? nd_jump_root+0xe5/0x160 [ 8860.384160] ? path_init+0x437/0x520 [ 8860.385215] path_openat+0xf7/0x500 [ 8860.386426] ? do_raw_spin_unlock+0x75/0x190 [ 8860.392413] do_filp_open+0x99/0x140 [ 8860.394785] ? getname_flags+0x6e/0x330 [ 8860.396864] ? __check_object_size+0xff/0x256 [ 8860.402730] ? do_raw_spin_unlock+0x75/0x190 [ 8860.404961] ? _raw_spin_unlock+0x12/0x30 [ 8860.405918] do_sys_openat2+0x2b4/0x410 [ 8860.408360] do_sys_open+0x73/0xa0 [ 8860.410482] __x64_sys_openat+0x24/0x30 [ 8860.414644] do_syscall_64+0xc1/0x440 [ 8860.416837] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8860.429400] RIP: 0033:0x7f1a63983332 [ 8860.431004] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8860.442543] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8860.448459] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8860.455950] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8860.459941] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8860.466826] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8860.473601] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8865.252471] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6842s! [ 8865.261870] Showing busy workqueues and worker pools: [ 8865.265642] workqueue events: flags=0x0 [ 8865.269810] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8865.274348] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8865.281905] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8865.294705] pending: sec_gc_main [ptlrpc] [ 8865.296617] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8865.307413] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8865.319929] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8865.326944] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8865.335209] workqueue events_long: flags=0x0 [ 8865.339084] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8865.348519] pending: lu_objects_flush [obdclass] [ 8865.353258] workqueue events_unbound: flags=0x2 [ 8865.359135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8865.362094] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8865.374080] workqueue events_power_efficient: flags=0x80 [ 8865.380108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8865.385679] pending: neigh_periodic_work, do_cache_clean [ 8865.388633] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8865.393903] pending: neigh_periodic_work, check_lifetime [ 8865.398563] workqueue mm_percpu_wq: flags=0x8 [ 8865.401824] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8865.405458] pending: vmstat_update [ 8865.410191] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8865.423118] pending: vmstat_update [ 8865.433127] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8865.439317] pending: vmstat_update [ 8865.440600] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8865.458535] pending: vmstat_update [ 8865.465579] workqueue writeback: flags=0x4a [ 8865.476090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8865.480725] pending: wb_workfn, wb_workfn [ 8865.488303] workqueue xprtiod: flags=0xa [ 8865.493434] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8865.504378] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8865.518473] workqueue ipv6_addrconf: flags=0x40008 [ 8865.531941] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8865.539912] pending: addrconf_verify_work [ 8865.544806] workqueue knbd0-recv: flags=0x1a [ 8865.557311] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8865.572127] in-flight: 647:recv_work [ 8865.574601] workqueue ptlrpc_pinger: flags=0x2 [ 8865.579806] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8865.587289] pending: ptlrpc_pinger_main [ptlrpc] [ 8865.589388] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8855s workers=2 idle: 652 [ 8872.059186] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8872.069115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8872.107258] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8872.115051] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8872.119175] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8872.121255] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8872.137189] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8872.148184] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8872.154672] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8872.171029] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8872.179179] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8872.190233] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8872.195323] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8872.199445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8872.204050] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8872.212918] Call Trace: [ 8872.215686] [ 8872.222969] ? show_regs.cold.9+0x22/0x2f [ 8872.225193] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8872.231236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8872.236602] ? watchdog+0x40/0x40 [ 8872.240655] ? hrtimer_interrupt+0x128/0x2c0 [ 8872.243960] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8872.246659] ? apic_timer_interrupt+0xf/0x20 [ 8872.249211] [ 8872.250379] ? native_safe_halt+0x2e/0x50 [ 8872.252366] kvm_wait+0xcd/0xe0 [ 8872.254275] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8872.256822] do_raw_spin_lock+0x177/0x190 [ 8872.259158] _raw_spin_lock+0x21/0x30 [ 8872.263651] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8872.273054] ? do_raw_spin_unlock+0x75/0x190 [ 8872.277418] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8872.280956] ? do_wait_intr+0xf0/0xf0 [ 8872.283252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8872.286081] mdc_close+0x53b/0xd70 [mdc] [ 8872.287694] lmv_close+0x18d/0x490 [lmv] [ 8872.289484] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8872.292044] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8872.294156] ll_md_close+0x274/0x800 [lustre] [ 8872.296286] ll_file_release+0x128/0xc50 [lustre] [ 8872.298044] __fput+0x10d/0x440 [ 8872.299478] ____fput+0x12/0x20 [ 8872.300749] task_work_run+0xc6/0x110 [ 8872.302387] exit_to_usermode_loop+0x1dd/0x1f0 [ 8872.304211] do_syscall_64+0x3d6/0x440 [ 8872.305978] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8872.307801] RIP: 0033:0x7f1a63983cd8 [ 8872.309176] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8872.315672] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8872.318263] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8872.320720] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8872.323447] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8872.326046] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8872.328478] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8880.056062] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 8880.058018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8880.059342] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8880.059415] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8880.059421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8880.059424] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8880.059437] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8880.059442] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8880.059446] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8880.059449] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8880.059451] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8880.059453] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8880.059455] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8880.059459] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8880.059461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8880.059464] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8880.059472] Call Trace: [ 8880.059477] [ 8880.059482] ? show_regs.cold.9+0x22/0x2f [ 8880.059491] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8880.059497] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8880.059501] ? watchdog+0x40/0x40 [ 8880.059505] ? hrtimer_interrupt+0x128/0x2c0 [ 8880.059508] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8880.059515] ? apic_timer_interrupt+0xf/0x20 [ 8880.059519] [ 8880.059521] ? native_safe_halt+0x2e/0x50 [ 8880.059525] kvm_wait+0xcd/0xe0 [ 8880.059531] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8880.059538] do_raw_spin_lock+0x177/0x190 [ 8880.059542] _raw_spin_lock+0x21/0x30 [ 8880.059545] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8880.060164] ? do_raw_spin_unlock+0x75/0x190 [ 8880.060171] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8880.070566] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8880.070643] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8880.070651] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8880.070654] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8880.070667] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8880.070671] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8880.070676] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8880.070678] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8880.070680] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8880.070682] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8880.070685] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8880.070688] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8880.070691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8880.070693] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8880.070702] Call Trace: [ 8880.070708] [ 8880.070714] ? show_regs.cold.9+0x22/0x2f [ 8880.070962] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8880.070969] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8880.070973] ? watchdog+0x40/0x40 [ 8880.070977] ? hrtimer_interrupt+0x128/0x2c0 [ 8880.070981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8880.070987] ? apic_timer_interrupt+0xf/0x20 [ 8880.071054] [ 8880.071056] ? native_safe_halt+0x2e/0x50 [ 8880.071059] kvm_wait+0xcd/0xe0 [ 8880.071064] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8880.071069] do_raw_spin_lock+0x177/0x190 [ 8880.071072] _raw_spin_lock+0x21/0x30 [ 8880.071077] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8880.071323] obd_mod_free+0x46/0x140 [ptlrpc] [ 8880.071519] mdc_commit_open+0x91/0xb0 [mdc] [ 8880.071554] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8880.072159] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8880.072379] after_reply+0xe2c/0x1780 [ptlrpc] [ 8880.072588] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8880.073246] ? __next_timer_interrupt+0x160/0x160 [ 8880.073257] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8880.073476] ? do_wait_intr+0xf0/0xf0 [ 8880.073485] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8880.073684] kthread+0x1d1/0x200 [ 8880.073690] ? set_kthread_struct+0x70/0x70 [ 8880.073694] ret_from_fork+0x1f/0x30 [ 8880.339494] ? do_wait_intr+0xf0/0xf0 [ 8880.343089] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8880.347214] mdc_close+0x53b/0xd70 [mdc] [ 8880.352181] lmv_close+0x18d/0x490 [lmv] [ 8880.354502] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8880.358952] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8880.362251] ll_md_close+0x274/0x800 [lustre] [ 8880.363970] ll_file_release+0x128/0xc50 [lustre] [ 8880.367078] __fput+0x10d/0x440 [ 8880.369212] ____fput+0x12/0x20 [ 8880.371326] task_work_run+0xc6/0x110 [ 8880.373807] exit_to_usermode_loop+0x1dd/0x1f0 [ 8880.377323] do_syscall_64+0x3d6/0x440 [ 8880.381104] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8880.383452] RIP: 0033:0x7f1a63983cd8 [ 8880.384765] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8880.391082] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8880.393712] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8880.396403] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8880.401474] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8880.405219] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8880.407510] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8892.053885] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 8892.060870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8892.111033] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8892.117846] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8892.126229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8892.129983] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8892.152971] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8892.160431] RAX: 0000000000007a86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8892.165771] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8892.170555] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8892.175467] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8892.180904] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8892.184194] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8892.188669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8892.190877] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8892.194566] Call Trace: [ 8892.196040] [ 8892.198000] ? show_regs.cold.9+0x22/0x2f [ 8892.202423] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8892.206414] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8892.208591] ? watchdog+0x40/0x40 [ 8892.210440] ? hrtimer_interrupt+0x128/0x2c0 [ 8892.217000] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8892.222239] ? apic_timer_interrupt+0xf/0x20 [ 8892.225029] [ 8892.226222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8892.230263] do_raw_spin_lock+0x177/0x190 [ 8892.233590] _raw_spin_lock+0x21/0x30 [ 8892.235356] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8892.242128] ? do_raw_spin_unlock+0x75/0x190 [ 8892.243670] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8892.249462] ? do_wait_intr+0xf0/0xf0 [ 8892.252934] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8892.256151] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8892.260321] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8892.263952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8892.267832] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8892.273008] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8892.275897] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8892.278067] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8892.283416] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8892.287221] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8892.292605] lmv_intent_open+0x338/0x1150 [lmv] [ 8892.295993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8892.300782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8892.306469] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8892.312278] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8892.315050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8892.322733] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8892.326767] ? do_raw_spin_unlock+0x75/0x190 [ 8892.329038] ? _raw_spin_unlock+0x12/0x30 [ 8892.332207] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8892.338982] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8892.345336] ll_atomic_open+0x36a/0xd70 [lustre] [ 8892.348996] lookup_open+0x356/0xb40 [ 8892.360158] do_last+0x313/0xfc0 [ 8892.363367] ? nd_jump_root+0xe5/0x160 [ 8892.367841] ? path_init+0x437/0x520 [ 8892.370979] path_openat+0xf7/0x500 [ 8892.374822] ? do_raw_spin_unlock+0x75/0x190 [ 8892.380745] do_filp_open+0x99/0x140 [ 8892.381961] ? getname_flags+0x6e/0x330 [ 8892.386334] ? __check_object_size+0xff/0x256 [ 8892.390648] ? do_raw_spin_unlock+0x75/0x190 [ 8892.395020] ? _raw_spin_unlock+0x12/0x30 [ 8892.398969] do_sys_openat2+0x2b4/0x410 [ 8892.402921] do_sys_open+0x73/0xa0 [ 8892.407637] __x64_sys_openat+0x24/0x30 [ 8892.413064] do_syscall_64+0xc1/0x440 [ 8892.415589] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8892.418482] RIP: 0033:0x7f1a63983332 [ 8892.420993] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8892.432992] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8892.439053] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8892.444412] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8892.449184] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8892.452903] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8892.457455] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8895.974165] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6872s! [ 8895.982872] Showing busy workqueues and worker pools: [ 8895.988143] workqueue events: flags=0x0 [ 8895.990274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8895.993596] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8895.999718] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8896.007840] pending: sec_gc_main [ptlrpc] [ 8896.015983] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8896.020980] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8896.028249] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8896.033961] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8896.044943] workqueue events_long: flags=0x0 [ 8896.049862] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8896.058811] pending: lu_objects_flush [obdclass] [ 8896.061982] workqueue events_unbound: flags=0x2 [ 8896.066263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8896.075070] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8896.084627] workqueue events_power_efficient: flags=0x80 [ 8896.090423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8896.101039] pending: neigh_periodic_work, do_cache_clean [ 8896.109326] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8896.115436] pending: neigh_periodic_work, check_lifetime [ 8896.123309] workqueue mm_percpu_wq: flags=0x8 [ 8896.129860] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8896.141618] pending: vmstat_update [ 8896.148828] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8896.160868] pending: vmstat_update [ 8896.166944] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8896.176796] pending: vmstat_update [ 8896.178792] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8896.185167] pending: vmstat_update [ 8896.191948] workqueue writeback: flags=0x4a [ 8896.195144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8896.201415] pending: wb_workfn, wb_workfn [ 8896.208311] workqueue xprtiod: flags=0xa [ 8896.214014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8896.223264] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8896.236837] workqueue ipv6_addrconf: flags=0x40008 [ 8896.243425] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8896.251793] pending: addrconf_verify_work [ 8896.259632] workqueue knbd0-recv: flags=0x1a [ 8896.267383] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8896.277799] in-flight: 647:recv_work [ 8896.282128] workqueue ptlrpc_pinger: flags=0x2 [ 8896.288150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8896.294281] pending: ptlrpc_pinger_main [ptlrpc] [ 8896.306034] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8886s workers=2 idle: 652 [ 8900.057066] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8900.061265] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8900.086860] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8900.093877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8900.098624] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8900.101804] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8900.110809] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8900.114821] RAX: 0000000000007a8d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8900.117881] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8900.121473] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8900.126952] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8900.130879] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8900.140734] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8900.145190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8900.152563] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8900.158316] Call Trace: [ 8900.159570] [ 8900.160734] ? show_regs.cold.9+0x22/0x2f [ 8900.162714] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8900.165243] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8900.169428] ? watchdog+0x40/0x40 [ 8900.171110] ? hrtimer_interrupt+0x128/0x2c0 [ 8900.173223] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8900.175580] ? apic_timer_interrupt+0xf/0x20 [ 8900.177268] [ 8900.178476] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8900.181014] do_raw_spin_lock+0x177/0x190 [ 8900.183005] _raw_spin_lock+0x21/0x30 [ 8900.184516] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8900.187151] ? do_raw_spin_unlock+0x75/0x190 [ 8900.189349] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8900.192230] ? do_wait_intr+0xf0/0xf0 [ 8900.194013] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8900.196680] mdc_close+0x53b/0xd70 [mdc] [ 8900.198116] lmv_close+0x18d/0x490 [lmv] [ 8900.199802] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8900.204508] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8900.209211] ll_md_close+0x274/0x800 [lustre] [ 8900.213170] ll_file_release+0x128/0xc50 [lustre] [ 8900.220165] __fput+0x10d/0x440 [ 8900.222442] ____fput+0x12/0x20 [ 8900.224895] task_work_run+0xc6/0x110 [ 8900.226731] exit_to_usermode_loop+0x1dd/0x1f0 [ 8900.228818] do_syscall_64+0x3d6/0x440 [ 8900.230227] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8900.232677] RIP: 0033:0x7f1a63983cd8 [ 8900.234436] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8900.246184] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8900.252199] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8900.255869] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8900.258914] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8900.262280] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8900.265294] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8908.060299] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 8908.062579] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 8908.069430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8908.069513] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8908.069538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8908.069540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8908.069555] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8908.069560] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8908.069565] RAX: 000000000000758f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8908.069567] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8908.069569] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8908.069571] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8908.069573] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8908.069576] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8908.069579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8908.069581] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8908.069589] Call Trace: [ 8908.069595] [ 8908.069602] ? show_regs.cold.9+0x22/0x2f [ 8908.069611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8908.069617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8908.069621] ? watchdog+0x40/0x40 [ 8908.069625] ? hrtimer_interrupt+0x128/0x2c0 [ 8908.069629] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8908.069638] ? apic_timer_interrupt+0xf/0x20 [ 8908.069642] [ 8908.069645] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8908.069651] do_raw_spin_lock+0x177/0x190 [ 8908.069656] _raw_spin_lock+0x21/0x30 [ 8908.069661] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8908.072704] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8908.072778] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8908.072785] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8908.072788] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8908.072800] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8908.072805] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8908.072810] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8908.072812] RDX: 00000000000055c2 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8908.072814] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8908.072817] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8908.072819] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8908.072822] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8908.072825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8908.072827] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8908.072836] Call Trace: [ 8908.072841] [ 8908.072847] ? show_regs.cold.9+0x22/0x2f [ 8908.072856] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8908.072862] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8908.072866] ? watchdog+0x40/0x40 [ 8908.072870] ? hrtimer_interrupt+0x128/0x2c0 [ 8908.072874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8908.072881] ? apic_timer_interrupt+0xf/0x20 [ 8908.072886] [ 8908.072889] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8908.072894] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8908.072899] do_raw_spin_lock+0x177/0x190 [ 8908.072904] _raw_spin_lock+0x21/0x30 [ 8908.072910] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8908.073178] obd_mod_free+0x46/0x140 [ptlrpc] [ 8908.147639] ? do_raw_spin_unlock+0x75/0x190 [ 8908.147659] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8908.147960] ? do_wait_intr+0xf0/0xf0 [ 8908.147969] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8908.153508] mdc_commit_open+0x91/0xb0 [mdc] [ 8908.153771] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8908.154039] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8908.154247] after_reply+0xe2c/0x1780 [ptlrpc] [ 8908.154631] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8908.154857] ? __next_timer_interrupt+0x160/0x160 [ 8908.154867] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8908.155100] ? do_wait_intr+0xf0/0xf0 [ 8908.155111] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8908.155622] kthread+0x1d1/0x200 [ 8908.155628] ? set_kthread_struct+0x70/0x70 [ 8908.155632] ret_from_fork+0x1f/0x30 [ 8908.641657] mdc_close+0x53b/0xd70 [mdc] [ 8908.645940] lmv_close+0x18d/0x490 [lmv] [ 8908.653132] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8908.663499] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8908.668245] ll_md_close+0x274/0x800 [lustre] [ 8908.677269] ll_file_release+0x128/0xc50 [lustre] [ 8908.686344] __fput+0x10d/0x440 [ 8908.687482] ____fput+0x12/0x20 [ 8908.695397] task_work_run+0xc6/0x110 [ 8908.696765] exit_to_usermode_loop+0x1dd/0x1f0 [ 8908.698380] do_syscall_64+0x3d6/0x440 [ 8908.710955] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8908.719010] RIP: 0033:0x7f1a63983cd8 [ 8908.724318] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8908.744222] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8908.752617] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8908.758012] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8908.768244] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8908.779838] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8908.784353] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8922.591017] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 8922.593698] rcu: 3-...!: (4354546 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 8922.597640] (detected by 2, t=6900189 jiffies, g=269633, q=2454) [ 8922.600184] Sending NMI from CPU 2 to CPUs 3: [ 8922.603558] NMI backtrace for cpu 3 [ 8922.603566] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8922.603568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8922.603570] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8922.603571] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8922.603573] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 8922.603576] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8922.603577] RDX: 0000000000007beb RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8922.603579] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8922.603580] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8922.603582] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8922.603584] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8922.603585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8922.603587] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8922.603588] Call Trace: [ 8922.603589] [ 8922.603590] ? show_regs.cold.9+0x22/0x2f [ 8922.603592] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8922.603593] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8922.603594] ? nmi_handle+0xbd/0x210 [ 8922.603595] ? default_do_nmi+0x4b/0x1b0 [ 8922.603596] ? do_nmi+0x280/0x380 [ 8922.603597] ? end_repeat_nmi+0x16/0x1f [ 8922.603599] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8922.603600] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8922.603602] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8922.603603] [ 8922.603604] do_raw_spin_lock+0x177/0x190 [ 8922.603606] _raw_spin_lock+0x21/0x30 [ 8922.603607] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8922.603608] obd_mod_free+0x46/0x140 [ptlrpc] [ 8922.603610] mdc_commit_open+0x91/0xb0 [mdc] [ 8922.603611] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8922.603612] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8922.603614] after_reply+0xe2c/0x1780 [ptlrpc] [ 8922.603615] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8922.603616] ? __next_timer_interrupt+0x160/0x160 [ 8922.603617] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8922.603618] ? do_wait_intr+0xf0/0xf0 [ 8922.603620] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8922.603621] kthread+0x1d1/0x200 [ 8922.603622] ? set_kthread_struct+0x70/0x70 [ 8922.603623] ret_from_fork+0x1f/0x30 [ 8922.605572] rcu: rcu_sched kthread starved for 6899844 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 8922.702853] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8922.706495] rcu: RCU grace-period kthread stack dump: [ 8922.708345] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8922.712593] Call Trace: [ 8922.713583] __schedule+0x351/0xcb0 [ 8922.714746] schedule+0xc0/0x180 [ 8922.716031] schedule_timeout+0xb4/0x190 [ 8922.717760] ? do_raw_spin_unlock+0x75/0x190 [ 8922.719347] ? __next_timer_interrupt+0x160/0x160 [ 8922.721089] ? prepare_to_swait_event+0xf3/0x140 [ 8922.723079] rcu_gp_kthread+0x8dd/0x15b0 [ 8922.724714] ? force_qs_rnp+0x2c0/0x2c0 [ 8922.726062] kthread+0x1d1/0x200 [ 8922.727184] ? set_kthread_struct+0x70/0x70 [ 8922.728880] ret_from_fork+0x1f/0x30 [ 8922.730418] rcu: Stack dump where RCU GP kthread last ran: [ 8922.732545] Sending NMI from CPU 2 to CPUs 1: [ 8922.735453] NMI backtrace for cpu 1 [ 8922.735458] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8922.735460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8922.735462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8922.735464] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8922.735466] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 [ 8922.735469] RAX: 00000000000073ec RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8922.735471] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8922.735472] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8922.735474] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8922.735475] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8922.735476] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8922.735478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8922.735479] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8922.735480] Call Trace: [ 8922.735481] [ 8922.735482] ? show_regs.cold.9+0x22/0x2f [ 8922.735483] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8922.735484] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8922.735486] ? nmi_handle+0xbd/0x210 [ 8922.735487] ? default_do_nmi+0x4b/0x1b0 [ 8922.735487] ? do_nmi+0x280/0x380 [ 8922.735488] ? end_repeat_nmi+0x16/0x1f [ 8922.735490] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8922.735491] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8922.735492] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8922.735493] [ 8922.735494] do_raw_spin_lock+0x177/0x190 [ 8922.735495] _raw_spin_lock+0x21/0x30 [ 8922.735496] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8922.735497] ? do_raw_spin_unlock+0x75/0x190 [ 8922.735499] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8922.735500] ? do_wait_intr+0xf0/0xf0 [ 8922.735501] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8922.735502] mdc_close+0x53b/0xd70 [mdc] [ 8922.735503] lmv_close+0x18d/0x490 [lmv] [ 8922.735504] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8922.735506] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8922.735507] ll_md_close+0x274/0x800 [lustre] [ 8922.735508] ll_file_release+0x128/0xc50 [lustre] [ 8922.735509] __fput+0x10d/0x440 [ 8922.735510] ____fput+0x12/0x20 [ 8922.735511] task_work_run+0xc6/0x110 [ 8922.735512] exit_to_usermode_loop+0x1dd/0x1f0 [ 8922.735513] do_syscall_64+0x3d6/0x440 [ 8922.735514] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8922.735516] RIP: 0033:0x7f1a63983cd8 [ 8922.735517] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8922.735519] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8922.735522] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8922.735523] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8922.735524] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8922.735526] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8922.735527] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8924.053776] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 8924.058077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8924.099243] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8924.107563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8924.111675] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8924.114134] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8924.125447] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8924.130873] RAX: 0000000000007108 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8924.133268] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8924.136082] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8924.139979] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8924.145897] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8924.149250] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8924.153790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8924.157804] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8924.161795] Call Trace: [ 8924.164115] [ 8924.165824] ? show_regs.cold.9+0x22/0x2f [ 8924.168536] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8924.171766] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8924.175211] ? watchdog+0x40/0x40 [ 8924.182209] ? hrtimer_interrupt+0x128/0x2c0 [ 8924.185828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8924.188982] ? apic_timer_interrupt+0xf/0x20 [ 8924.191312] [ 8924.193398] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8924.199774] do_raw_spin_lock+0x177/0x190 [ 8924.202911] _raw_spin_lock+0x21/0x30 [ 8924.205729] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8924.210707] ? do_raw_spin_unlock+0x75/0x190 [ 8924.212229] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8924.220016] ? do_wait_intr+0xf0/0xf0 [ 8924.222708] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8924.228668] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8924.232246] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8924.237773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8924.240950] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8924.246243] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8924.249456] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8924.253001] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8924.256058] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8924.260775] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8924.264373] lmv_intent_open+0x338/0x1150 [lmv] [ 8924.266347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8924.268413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8924.270962] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8924.276547] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8924.280284] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8924.284751] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8924.287290] ? do_raw_spin_unlock+0x75/0x190 [ 8924.289855] ? _raw_spin_unlock+0x12/0x30 [ 8924.292186] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8924.293635] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8924.295649] ll_atomic_open+0x36a/0xd70 [lustre] [ 8924.298658] lookup_open+0x356/0xb40 [ 8924.300692] do_last+0x313/0xfc0 [ 8924.302176] ? nd_jump_root+0xe5/0x160 [ 8924.304060] ? path_init+0x437/0x520 [ 8924.306787] path_openat+0xf7/0x500 [ 8924.308321] ? do_raw_spin_unlock+0x75/0x190 [ 8924.310000] do_filp_open+0x99/0x140 [ 8924.311241] ? getname_flags+0x6e/0x330 [ 8924.313106] ? __check_object_size+0xff/0x256 [ 8924.315443] ? do_raw_spin_unlock+0x75/0x190 [ 8924.317233] ? _raw_spin_unlock+0x12/0x30 [ 8924.319237] do_sys_openat2+0x2b4/0x410 [ 8924.321185] do_sys_open+0x73/0xa0 [ 8924.322854] __x64_sys_openat+0x24/0x30 [ 8924.324850] do_syscall_64+0xc1/0x440 [ 8924.326538] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8924.329258] RIP: 0033:0x7f1a63983332 [ 8924.330833] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8924.339241] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8924.342556] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8924.345812] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8924.349662] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8924.354086] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8924.360697] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8926.689028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6903s! [ 8926.692169] Showing busy workqueues and worker pools: [ 8926.694122] workqueue events: flags=0x0 [ 8926.695422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8926.697921] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8926.700594] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8926.703173] pending: sec_gc_main [ptlrpc] [ 8926.705349] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8926.707654] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8926.711728] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8926.714019] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8926.717489] workqueue events_long: flags=0x0 [ 8926.719129] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8926.721565] pending: lu_objects_flush [obdclass] [ 8926.723654] workqueue events_unbound: flags=0x2 [ 8926.725349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8926.727496] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8926.731549] workqueue events_power_efficient: flags=0x80 [ 8926.733320] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8926.735296] pending: neigh_periodic_work, do_cache_clean [ 8926.737321] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8926.739700] pending: neigh_periodic_work, check_lifetime [ 8926.741301] workqueue mm_percpu_wq: flags=0x8 [ 8926.742859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8926.744748] pending: vmstat_update [ 8926.746682] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8926.749113] pending: vmstat_update [ 8926.750329] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8926.752754] pending: vmstat_update [ 8926.754049] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8926.756470] pending: vmstat_update [ 8926.757742] workqueue writeback: flags=0x4a [ 8926.759509] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8926.761915] pending: wb_workfn, wb_workfn [ 8926.763694] workqueue xprtiod: flags=0xa [ 8926.764925] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8926.767257] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8926.770370] workqueue ipv6_addrconf: flags=0x40008 [ 8926.773319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8926.775860] pending: addrconf_verify_work [ 8926.777673] workqueue knbd0-recv: flags=0x1a [ 8926.778967] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8926.781762] in-flight: 647:recv_work [ 8926.783195] workqueue ptlrpc_pinger: flags=0x2 [ 8926.784718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8926.786780] pending: ptlrpc_pinger_main [ptlrpc] [ 8926.789215] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8916s workers=2 idle: 652 [ 8948.056801] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8948.058030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8948.060606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 8948.063569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8948.063804] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8948.063812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8948.063815] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8948.063840] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8948.063844] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8948.063848] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8948.063851] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8948.063853] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8948.063854] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8948.063856] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8948.063860] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8948.063863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8948.063865] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8948.063873] Call Trace: [ 8948.063891] [ 8948.063898] ? show_regs.cold.9+0x22/0x2f [ 8948.063909] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8948.063916] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8948.063920] ? watchdog+0x40/0x40 [ 8948.063924] ? hrtimer_interrupt+0x128/0x2c0 [ 8948.063928] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8948.063933] ? apic_timer_interrupt+0xf/0x20 [ 8948.063938] [ 8948.063941] ? native_safe_halt+0x2e/0x50 [ 8948.063944] kvm_wait+0xcd/0xe0 [ 8948.063951] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8948.063959] do_raw_spin_lock+0x177/0x190 [ 8948.063963] _raw_spin_lock+0x21/0x30 [ 8948.063968] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8948.065031] obd_mod_free+0x46/0x140 [ptlrpc] [ 8948.066779] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8948.066835] mdc_commit_open+0x91/0xb0 [mdc] [ 8948.066881] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8948.067120] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8948.067340] after_reply+0xe2c/0x1780 [ptlrpc] [ 8948.068902] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8948.069114] ? __next_timer_interrupt+0x160/0x160 [ 8948.069141] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8948.069531] ? do_wait_intr+0xf0/0xf0 [ 8948.069540] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8948.069884] kthread+0x1d1/0x200 [ 8948.069891] ? set_kthread_struct+0x70/0x70 [ 8948.069895] ret_from_fork+0x1f/0x30 [ 8948.071958] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8948.072030] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8948.072037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8948.072041] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8948.072054] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8948.072058] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8948.072062] RAX: 00000000000007bb RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8948.072065] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8948.072067] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8948.072070] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8948.072072] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8948.072076] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8948.072079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8948.072081] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8948.072089] Call Trace: [ 8948.072094] [ 8948.072099] ? show_regs.cold.9+0x22/0x2f [ 8948.072108] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8948.072114] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8948.072118] ? watchdog+0x40/0x40 [ 8948.072123] ? hrtimer_interrupt+0x128/0x2c0 [ 8948.072144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8948.072152] ? apic_timer_interrupt+0xf/0x20 [ 8948.072157] [ 8948.072160] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8948.072167] do_raw_spin_lock+0x177/0x190 [ 8948.072171] _raw_spin_lock+0x21/0x30 [ 8948.072176] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8948.072573] ? do_raw_spin_unlock+0x75/0x190 [ 8948.072579] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8948.072844] ? do_wait_intr+0xf0/0xf0 [ 8948.072852] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8948.073055] mdc_close+0x53b/0xd70 [mdc] [ 8948.073096] lmv_close+0x18d/0x490 [lmv] [ 8948.073194] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8948.073365] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8948.073890] ll_md_close+0x274/0x800 [lustre] [ 8948.073978] ll_file_release+0x128/0xc50 [lustre] [ 8948.074061] __fput+0x10d/0x440 [ 8948.074070] ____fput+0x12/0x20 [ 8948.074074] task_work_run+0xc6/0x110 [ 8948.074082] exit_to_usermode_loop+0x1dd/0x1f0 [ 8948.074091] do_syscall_64+0x3d6/0x440 [ 8948.074096] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8948.074103] RIP: 0033:0x7f1a63983cd8 [ 8948.074249] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8948.074256] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8948.074262] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8948.074264] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8948.074266] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8948.074268] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8948.074270] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8948.409578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8948.431682] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8948.436537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8948.439985] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8948.442479] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8948.449624] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8948.452429] RAX: 0000000000005e8b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8948.454809] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8948.457642] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8948.460026] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8948.462593] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8948.465424] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8948.468092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8948.470428] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8948.473106] Call Trace: [ 8948.474345] [ 8948.474877] ? show_regs.cold.9+0x22/0x2f [ 8948.476488] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8948.478115] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8948.479882] ? watchdog+0x40/0x40 [ 8948.480871] ? hrtimer_interrupt+0x128/0x2c0 [ 8948.482201] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8948.484003] ? apic_timer_interrupt+0xf/0x20 [ 8948.485522] [ 8948.486404] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8948.488555] do_raw_spin_lock+0x177/0x190 [ 8948.490116] _raw_spin_lock+0x21/0x30 [ 8948.491554] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8948.494497] ? do_raw_spin_unlock+0x75/0x190 [ 8948.495852] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8948.498024] ? do_wait_intr+0xf0/0xf0 [ 8948.499794] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8948.501994] mdc_close+0x53b/0xd70 [mdc] [ 8948.503459] lmv_close+0x18d/0x490 [lmv] [ 8948.504969] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8948.507533] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8948.509371] ll_md_close+0x274/0x800 [lustre] [ 8948.510882] ll_file_release+0x128/0xc50 [lustre] [ 8948.512931] __fput+0x10d/0x440 [ 8948.513960] ____fput+0x12/0x20 [ 8948.515478] task_work_run+0xc6/0x110 [ 8948.516910] exit_to_usermode_loop+0x1dd/0x1f0 [ 8948.518563] do_syscall_64+0x3d6/0x440 [ 8948.520193] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8948.522476] RIP: 0033:0x7f1a63983cd8 [ 8948.523866] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8948.530633] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8948.533551] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8948.536300] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8948.539087] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8948.541659] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8948.544671] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8952.053441] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 8952.056781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8952.104798] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8952.114846] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8952.123011] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8952.129932] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8952.148904] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8952.155702] RAX: 0000000000000086 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8952.161215] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8952.169964] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8952.175068] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8952.181715] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8952.186858] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8952.194153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8952.198031] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8952.205505] Call Trace: [ 8952.207396] [ 8952.210674] ? show_regs.cold.9+0x22/0x2f [ 8952.214349] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8952.218560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8952.223306] ? watchdog+0x40/0x40 [ 8952.224955] ? hrtimer_interrupt+0x128/0x2c0 [ 8952.227002] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8952.230348] ? apic_timer_interrupt+0xf/0x20 [ 8952.233630] [ 8952.237169] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8952.240332] do_raw_spin_lock+0x177/0x190 [ 8952.242050] _raw_spin_lock+0x21/0x30 [ 8952.248046] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8952.253238] ? do_raw_spin_unlock+0x75/0x190 [ 8952.256768] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8952.262317] ? do_wait_intr+0xf0/0xf0 [ 8952.266442] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8952.271839] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8952.277486] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8952.284342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8952.287880] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8952.293045] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8952.295982] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8952.298106] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8952.300555] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8952.304000] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8952.309298] lmv_intent_open+0x338/0x1150 [lmv] [ 8952.312327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8952.320706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8952.324371] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8952.329259] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8952.334659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8952.339285] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8952.342194] ? do_raw_spin_unlock+0x75/0x190 [ 8952.344257] ? _raw_spin_unlock+0x12/0x30 [ 8952.349713] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8952.356657] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8952.362547] ll_atomic_open+0x36a/0xd70 [lustre] [ 8952.369572] lookup_open+0x356/0xb40 [ 8952.371375] do_last+0x313/0xfc0 [ 8952.376342] ? nd_jump_root+0xe5/0x160 [ 8952.379778] ? path_init+0x437/0x520 [ 8952.384785] path_openat+0xf7/0x500 [ 8952.388605] ? do_raw_spin_unlock+0x75/0x190 [ 8952.390315] do_filp_open+0x99/0x140 [ 8952.395570] ? getname_flags+0x6e/0x330 [ 8952.397120] ? __check_object_size+0xff/0x256 [ 8952.400946] ? do_raw_spin_unlock+0x75/0x190 [ 8952.404547] ? _raw_spin_unlock+0x12/0x30 [ 8952.406243] do_sys_openat2+0x2b4/0x410 [ 8952.409271] do_sys_open+0x73/0xa0 [ 8952.411034] __x64_sys_openat+0x24/0x30 [ 8952.412820] do_syscall_64+0xc1/0x440 [ 8952.414603] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8952.416666] RIP: 0033:0x7f1a63983332 [ 8952.420631] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8952.435972] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8952.439708] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8952.443043] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8952.446502] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8952.449485] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8952.456345] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8956.896874] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6933s! [ 8956.908998] Showing busy workqueues and worker pools: [ 8956.913241] workqueue events: flags=0x0 [ 8956.923354] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8956.928864] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8956.940182] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8956.946957] pending: sec_gc_main [ptlrpc] [ 8956.950203] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8956.957863] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8956.972942] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8956.987095] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8957.002363] workqueue events_long: flags=0x0 [ 8957.006474] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8957.016792] pending: lu_objects_flush [obdclass] [ 8957.024968] workqueue events_unbound: flags=0x2 [ 8957.029566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8957.034700] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8957.048420] workqueue events_power_efficient: flags=0x80 [ 8957.052694] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8957.067999] pending: neigh_periodic_work, do_cache_clean [ 8957.071955] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8957.081941] pending: neigh_periodic_work, check_lifetime [ 8957.088335] workqueue mm_percpu_wq: flags=0x8 [ 8957.096501] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8957.102549] pending: vmstat_update [ 8957.107810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8957.117587] pending: vmstat_update [ 8957.122757] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8957.128175] pending: vmstat_update [ 8957.132035] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8957.148993] pending: vmstat_update [ 8957.151978] workqueue writeback: flags=0x4a [ 8957.159613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8957.173826] pending: wb_workfn, wb_workfn [ 8957.183544] workqueue xprtiod: flags=0xa [ 8957.191153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8957.199175] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8957.206945] workqueue ipv6_addrconf: flags=0x40008 [ 8957.218586] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8957.224217] pending: addrconf_verify_work [ 8957.230394] workqueue knbd0-recv: flags=0x1a [ 8957.233019] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8957.247769] in-flight: 647:recv_work [ 8957.257463] workqueue ptlrpc_pinger: flags=0x2 [ 8957.264689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8957.272545] pending: ptlrpc_pinger_main [ptlrpc] [ 8957.281562] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8947s workers=2 idle: 652 [ 8976.056379] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 8976.057038] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 8976.058079] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 8976.058086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8976.058160] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8976.058164] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8976.058167] RIP: 0010:__pv_queued_spin_lock_slowpath+0x185/0x6d0 [ 8976.058181] Code: 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 48 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 <0f> b6 03 84 c0 75 e6 48 83 05 2c d6 dc 02 01 44 89 f8 f0 66 44 0f [ 8976.058184] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8976.058189] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 8976.058192] RDX: 0000000000006109 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 8976.058194] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 8976.058196] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 8976.058199] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8976.058201] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 8976.058204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8976.058206] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 8976.058215] Call Trace: [ 8976.058220] [ 8976.058224] ? show_regs.cold.9+0x22/0x2f [ 8976.058234] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8976.058240] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8976.058244] ? watchdog+0x40/0x40 [ 8976.058248] ? hrtimer_interrupt+0x128/0x2c0 [ 8976.058251] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8976.058259] ? apic_timer_interrupt+0xf/0x20 [ 8976.058264] [ 8976.058266] ? __pv_queued_spin_lock_slowpath+0x185/0x6d0 [ 8976.058270] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8976.058275] do_raw_spin_lock+0x177/0x190 [ 8976.058279] _raw_spin_lock+0x21/0x30 [ 8976.058285] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8976.058568] obd_mod_free+0x46/0x140 [ptlrpc] [ 8976.058793] mdc_commit_open+0x91/0xb0 [mdc] [ 8976.058834] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8976.059073] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8976.059303] after_reply+0xe2c/0x1780 [ptlrpc] [ 8976.059532] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8976.059763] ? __next_timer_interrupt+0x160/0x160 [ 8976.059772] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8976.062133] ? do_wait_intr+0xf0/0xf0 [ 8976.062144] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8976.063316] kthread+0x1d1/0x200 [ 8976.063638] ? set_kthread_struct+0x70/0x70 [ 8976.063924] ret_from_fork+0x1f/0x30 [ 8976.071911] Modules linked in: [ 8976.074108] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8976.074183] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8976.074190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8976.074192] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8976.074206] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8976.074210] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8976.074213] RAX: 0000000000007586 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8976.074216] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 8976.074218] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 8976.074220] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 8976.074222] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 8976.074225] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 8976.074227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8976.074229] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 8976.074238] Call Trace: [ 8976.074242] [ 8976.074248] ? show_regs.cold.9+0x22/0x2f [ 8976.074256] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8976.074263] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8976.074267] ? watchdog+0x40/0x40 [ 8976.074271] ? hrtimer_interrupt+0x128/0x2c0 [ 8976.074274] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8976.074282] ? apic_timer_interrupt+0xf/0x20 [ 8976.074286] [ 8976.074289] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8976.074295] do_raw_spin_lock+0x177/0x190 [ 8976.074299] _raw_spin_lock+0x21/0x30 [ 8976.074304] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8976.074711] ? do_raw_spin_unlock+0x75/0x190 [ 8976.074717] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8976.074997] ? do_wait_intr+0xf0/0xf0 [ 8976.075005] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8976.075221] mdc_close+0x53b/0xd70 [mdc] [ 8976.075265] lmv_close+0x18d/0x490 [lmv] [ 8976.075295] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8976.075412] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8976.075489] ll_md_close+0x274/0x800 [lustre] [ 8976.075719] ll_file_release+0x128/0xc50 [lustre] [ 8976.075878] __fput+0x10d/0x440 [ 8976.075887] ____fput+0x12/0x20 [ 8976.075891] task_work_run+0xc6/0x110 [ 8976.075899] exit_to_usermode_loop+0x1dd/0x1f0 [ 8976.075907] do_syscall_64+0x3d6/0x440 [ 8976.075911] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8976.075918] RIP: 0033:0x7f1a63983cd8 [ 8976.075925] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8976.075929] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8976.075934] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 8976.075936] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8976.075939] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 8976.075941] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 8976.075942] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8976.792016] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8976.846555] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8976.865747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8976.884642] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8976.889997] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8976.903640] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8976.908045] RAX: 0000000000007f84 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8976.914686] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 8976.918930] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8976.929841] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 8976.938154] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 8976.953178] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 8976.964839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8976.982113] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 8976.989919] Call Trace: [ 8976.993837] [ 8976.996959] ? show_regs.cold.9+0x22/0x2f [ 8977.007443] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8977.014346] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8977.020080] ? watchdog+0x40/0x40 [ 8977.022018] ? hrtimer_interrupt+0x128/0x2c0 [ 8977.030305] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8977.034803] ? apic_timer_interrupt+0xf/0x20 [ 8977.044028] [ 8977.045528] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8977.049165] do_raw_spin_lock+0x177/0x190 [ 8977.051914] _raw_spin_lock+0x21/0x30 [ 8977.054423] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8977.059343] ? do_raw_spin_unlock+0x75/0x190 [ 8977.062389] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8977.066626] ? do_wait_intr+0xf0/0xf0 [ 8977.069055] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8977.074264] mdc_close+0x53b/0xd70 [mdc] [ 8977.076840] lmv_close+0x18d/0x490 [lmv] [ 8977.078964] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8977.082038] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8977.084798] ll_md_close+0x274/0x800 [lustre] [ 8977.087737] ll_file_release+0x128/0xc50 [lustre] [ 8977.089615] __fput+0x10d/0x440 [ 8977.090922] ____fput+0x12/0x20 [ 8977.098743] task_work_run+0xc6/0x110 [ 8977.101206] exit_to_usermode_loop+0x1dd/0x1f0 [ 8977.103177] do_syscall_64+0x3d6/0x440 [ 8977.105017] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8977.106825] RIP: 0033:0x7f1a63983cd8 [ 8977.109810] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8977.116149] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8977.119010] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 8977.121294] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 8977.123260] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 8977.126550] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 8977.129087] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8984.053123] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 8984.057734] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8984.113474] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8984.125009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8984.139768] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8984.145330] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8984.174445] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8984.183877] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 8984.192014] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 8984.196925] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 8984.201578] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 8984.204984] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 8984.209397] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 8984.214023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8984.215857] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 8984.219987] Call Trace: [ 8984.221454] [ 8984.222429] ? show_regs.cold.9+0x22/0x2f [ 8984.224370] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8984.226452] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8984.233503] ? watchdog+0x40/0x40 [ 8984.235278] ? hrtimer_interrupt+0x128/0x2c0 [ 8984.241645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8984.247630] ? apic_timer_interrupt+0xf/0x20 [ 8984.252917] [ 8984.253691] ? native_safe_halt+0x2e/0x50 [ 8984.261192] kvm_wait+0xcd/0xe0 [ 8984.266175] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8984.276740] do_raw_spin_lock+0x177/0x190 [ 8984.278218] _raw_spin_lock+0x21/0x30 [ 8984.286966] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8984.290666] ? do_raw_spin_unlock+0x75/0x190 [ 8984.301024] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8984.307281] ? do_wait_intr+0xf0/0xf0 [ 8984.316338] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8984.332027] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8984.342712] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8984.352365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8984.357600] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8984.367467] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8984.374360] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8984.381441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8984.388387] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8984.399278] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8984.409541] lmv_intent_open+0x338/0x1150 [lmv] [ 8984.414728] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8984.426762] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8984.435261] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8984.442274] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8984.446394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8984.455304] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8984.460366] ? do_raw_spin_unlock+0x75/0x190 [ 8984.469040] ? _raw_spin_unlock+0x12/0x30 [ 8984.476663] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8984.478644] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8984.489023] ll_atomic_open+0x36a/0xd70 [lustre] [ 8984.499407] lookup_open+0x356/0xb40 [ 8984.503971] do_last+0x313/0xfc0 [ 8984.511985] ? nd_jump_root+0xe5/0x160 [ 8984.520350] ? path_init+0x437/0x520 [ 8984.521633] path_openat+0xf7/0x500 [ 8984.522718] ? do_raw_spin_unlock+0x75/0x190 [ 8984.532643] do_filp_open+0x99/0x140 [ 8984.534487] ? getname_flags+0x6e/0x330 [ 8984.540626] ? __check_object_size+0xff/0x256 [ 8984.544067] ? do_raw_spin_unlock+0x75/0x190 [ 8984.549996] ? _raw_spin_unlock+0x12/0x30 [ 8984.553052] do_sys_openat2+0x2b4/0x410 [ 8984.561104] do_sys_open+0x73/0xa0 [ 8984.562419] __x64_sys_openat+0x24/0x30 [ 8984.565651] do_syscall_64+0xc1/0x440 [ 8984.571606] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8984.579030] RIP: 0033:0x7f1a63983332 [ 8984.581726] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8984.613588] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8984.624813] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 8984.640772] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 8984.652827] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 8984.659865] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 8984.669462] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 8987.616236] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6964s! [ 8987.628044] Showing busy workqueues and worker pools: [ 8987.629687] workqueue events: flags=0x0 [ 8987.631168] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8987.633225] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 8987.636134] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8987.638708] pending: sec_gc_main [ptlrpc] [ 8987.640470] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8987.642808] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 8987.646440] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8987.648606] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 8987.652729] workqueue events_long: flags=0x0 [ 8987.655409] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8987.658268] pending: lu_objects_flush [obdclass] [ 8987.660591] workqueue events_unbound: flags=0x2 [ 8987.662765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8987.665390] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8987.669581] workqueue events_power_efficient: flags=0x80 [ 8987.672086] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8987.674712] pending: neigh_periodic_work, do_cache_clean [ 8987.677308] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8987.680247] pending: neigh_periodic_work, check_lifetime [ 8987.682572] workqueue mm_percpu_wq: flags=0x8 [ 8987.684594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8987.687285] pending: vmstat_update [ 8987.688642] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8987.691150] pending: vmstat_update [ 8987.692375] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8987.694730] pending: vmstat_update [ 8987.696261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8987.699066] pending: vmstat_update [ 8987.700433] workqueue writeback: flags=0x4a [ 8987.701825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8987.704176] pending: wb_workfn, wb_workfn [ 8987.705549] workqueue xprtiod: flags=0xa [ 8987.707072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8987.709374] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 8987.712427] workqueue ipv6_addrconf: flags=0x40008 [ 8987.714249] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8987.716567] pending: addrconf_verify_work [ 8987.718277] workqueue knbd0-recv: flags=0x1a [ 8987.719656] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8987.722131] in-flight: 647:recv_work [ 8987.723508] workqueue ptlrpc_pinger: flags=0x2 [ 8987.725279] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8987.727659] pending: ptlrpc_pinger_main [ptlrpc] [ 8987.730190] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8977s workers=2 idle: 652 [ 9004.056021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9004.057022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9004.058033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9004.058042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9004.058095] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9004.058100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9004.058102] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9004.058113] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9004.058116] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9004.058120] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9004.058122] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9004.058123] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9004.058125] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9004.058126] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9004.058128] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9004.058130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9004.058131] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9004.058138] Call Trace: [ 9004.058143] [ 9004.058147] ? show_regs.cold.9+0x22/0x2f [ 9004.058154] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9004.058158] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9004.058162] ? watchdog+0x40/0x40 [ 9004.058167] ? hrtimer_interrupt+0x128/0x2c0 [ 9004.058171] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9004.058178] ? apic_timer_interrupt+0xf/0x20 [ 9004.058182] [ 9004.058184] ? native_safe_halt+0x2e/0x50 [ 9004.058188] kvm_wait+0xcd/0xe0 [ 9004.058198] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9004.058206] do_raw_spin_lock+0x177/0x190 [ 9004.058210] _raw_spin_lock+0x21/0x30 [ 9004.058214] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9004.058843] obd_mod_free+0x46/0x140 [ptlrpc] [ 9004.059036] mdc_commit_open+0x91/0xb0 [mdc] [ 9004.059074] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9004.059266] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9004.059460] after_reply+0xe2c/0x1780 [ptlrpc] [ 9004.059728] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9004.059920] ? __next_timer_interrupt+0x160/0x160 [ 9004.059928] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9004.060028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9004.060097] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9004.060103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9004.060106] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9004.060117] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9004.060122] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9004.060132] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9004.060135] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9004.060131] ? do_wait_intr+0xf0/0xf0 [ 9004.060137] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9004.060144] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9004.060148] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9004.060152] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9004.060154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9004.060157] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9004.060166] Call Trace: [ 9004.060170] [ 9004.060173] ? show_regs.cold.9+0x22/0x2f [ 9004.060182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9004.060188] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9004.060192] ? watchdog+0x40/0x40 [ 9004.060196] ? hrtimer_interrupt+0x128/0x2c0 [ 9004.060200] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9004.060208] ? apic_timer_interrupt+0xf/0x20 [ 9004.060213] [ 9004.060215] ? native_safe_halt+0x2e/0x50 [ 9004.060219] kvm_wait+0xcd/0xe0 [ 9004.060227] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9004.060236] do_raw_spin_lock+0x177/0x190 [ 9004.060241] _raw_spin_lock+0x21/0x30 [ 9004.060246] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9004.060146] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9004.060769] kthread+0x1d1/0x200 [ 9004.060774] ? set_kthread_struct+0x70/0x70 [ 9004.060778] ret_from_fork+0x1f/0x30 [ 9004.066833] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [ 9004.070554] ? do_raw_spin_unlock+0x75/0x190 [ 9004.114393] sha512_generic [ 9004.114407] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9004.114415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9004.114418] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9004.114431] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9004.114434] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9004.114440] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9004.114442] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9004.114444] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9004.114446] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9004.114449] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9004.114453] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9004.114456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9004.114458] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9004.114468] Call Trace: [ 9004.114475] [ 9004.114482] ? show_regs.cold.9+0x22/0x2f [ 9004.114493] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9004.114500] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9004.114504] ? watchdog+0x40/0x40 [ 9004.114509] ? hrtimer_interrupt+0x128/0x2c0 [ 9004.114513] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9004.114520] ? apic_timer_interrupt+0xf/0x20 [ 9004.114525] [ 9004.114528] ? native_safe_halt+0x2e/0x50 [ 9004.114532] kvm_wait+0xcd/0xe0 [ 9004.114540] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9004.114550] do_raw_spin_lock+0x177/0x190 [ 9004.114555] _raw_spin_lock+0x21/0x30 [ 9004.114559] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9004.114910] ? do_raw_spin_unlock+0x75/0x190 [ 9004.114917] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9004.115189] ? do_wait_intr+0xf0/0xf0 [ 9004.115198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9004.115405] mdc_close+0x53b/0xd70 [mdc] [ 9004.115444] lmv_close+0x18d/0x490 [lmv] [ 9004.115472] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9004.115583] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9004.116290] ll_md_close+0x274/0x800 [lustre] [ 9004.116378] ll_file_release+0x128/0xc50 [lustre] [ 9004.116462] __fput+0x10d/0x440 [ 9004.116470] ____fput+0x12/0x20 [ 9004.116474] task_work_run+0xc6/0x110 [ 9004.116481] exit_to_usermode_loop+0x1dd/0x1f0 [ 9004.116488] do_syscall_64+0x3d6/0x440 [ 9004.116493] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9004.116500] RIP: 0033:0x7f1a63983cd8 [ 9004.116507] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9004.116511] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9004.116516] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9004.116519] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9004.116521] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9004.116524] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9004.116526] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9004.715830] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9004.721281] ? do_wait_intr+0xf0/0xf0 [ 9004.723757] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9004.726554] mdc_close+0x53b/0xd70 [mdc] [ 9004.730190] lmv_close+0x18d/0x490 [lmv] [ 9004.735713] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9004.739189] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9004.745611] ll_md_close+0x274/0x800 [lustre] [ 9004.754677] ll_file_release+0x128/0xc50 [lustre] [ 9004.760974] __fput+0x10d/0x440 [ 9004.762300] ____fput+0x12/0x20 [ 9004.766206] task_work_run+0xc6/0x110 [ 9004.768657] exit_to_usermode_loop+0x1dd/0x1f0 [ 9004.771219] do_syscall_64+0x3d6/0x440 [ 9004.774856] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9004.779171] RIP: 0033:0x7f1a63983cd8 [ 9004.782094] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9004.791135] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9004.794053] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9004.800857] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9004.805529] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9004.812752] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9004.823732] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9012.053059] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 9012.058426] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9012.095390] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9012.103319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9012.110623] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9012.116389] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9012.137879] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9012.143444] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9012.146815] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9012.150864] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9012.154762] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9012.158876] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9012.162009] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9012.167021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9012.175137] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9012.179164] Call Trace: [ 9012.182233] [ 9012.183554] ? show_regs.cold.9+0x22/0x2f [ 9012.187830] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9012.191228] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9012.193620] ? watchdog+0x40/0x40 [ 9012.194570] ? hrtimer_interrupt+0x128/0x2c0 [ 9012.196069] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9012.198785] ? apic_timer_interrupt+0xf/0x20 [ 9012.203154] [ 9012.204048] ? native_safe_halt+0x2e/0x50 [ 9012.207167] kvm_wait+0xcd/0xe0 [ 9012.210643] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9012.213582] do_raw_spin_lock+0x177/0x190 [ 9012.215082] _raw_spin_lock+0x21/0x30 [ 9012.218074] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9012.220123] ? do_raw_spin_unlock+0x75/0x190 [ 9012.222426] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9012.227061] ? do_wait_intr+0xf0/0xf0 [ 9012.228244] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9012.230127] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9012.232654] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9012.235182] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9012.237303] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9012.239276] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9012.242207] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9012.244947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9012.249029] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9012.252202] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9012.256053] lmv_intent_open+0x338/0x1150 [lmv] [ 9012.264029] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9012.267388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9012.269982] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9012.271824] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9012.276466] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9012.283918] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9012.285741] ? do_raw_spin_unlock+0x75/0x190 [ 9012.287254] ? _raw_spin_unlock+0x12/0x30 [ 9012.290375] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9012.293714] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9012.296595] ll_atomic_open+0x36a/0xd70 [lustre] [ 9012.299360] lookup_open+0x356/0xb40 [ 9012.301089] do_last+0x313/0xfc0 [ 9012.302486] ? nd_jump_root+0xe5/0x160 [ 9012.307194] ? path_init+0x437/0x520 [ 9012.309497] path_openat+0xf7/0x500 [ 9012.313921] ? do_raw_spin_unlock+0x75/0x190 [ 9012.317123] do_filp_open+0x99/0x140 [ 9012.319264] ? getname_flags+0x6e/0x330 [ 9012.321073] ? __check_object_size+0xff/0x256 [ 9012.325396] ? do_raw_spin_unlock+0x75/0x190 [ 9012.336421] ? _raw_spin_unlock+0x12/0x30 [ 9012.341364] do_sys_openat2+0x2b4/0x410 [ 9012.347199] do_sys_open+0x73/0xa0 [ 9012.352617] __x64_sys_openat+0x24/0x30 [ 9012.354032] do_syscall_64+0xc1/0x440 [ 9012.365145] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9012.371454] RIP: 0033:0x7f1a63983332 [ 9012.374590] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9012.383646] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9012.386925] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9012.390467] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9012.397866] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9012.402860] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9012.415785] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9017.826505] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6994s! [ 9017.830828] Showing busy workqueues and worker pools: [ 9017.833182] workqueue events: flags=0x0 [ 9017.834645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9017.837286] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9017.840810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9017.843013] pending: sec_gc_main [ptlrpc] [ 9017.845231] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9017.847943] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9017.852029] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9017.854752] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9017.859686] workqueue events_long: flags=0x0 [ 9017.862090] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9017.868547] pending: lu_objects_flush [obdclass] [ 9017.875448] workqueue events_unbound: flags=0x2 [ 9017.877891] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9017.881123] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9017.888113] workqueue events_power_efficient: flags=0x80 [ 9017.890048] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9017.899793] pending: neigh_periodic_work, do_cache_clean [ 9017.907168] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9017.917645] pending: neigh_periodic_work, check_lifetime [ 9017.927697] workqueue mm_percpu_wq: flags=0x8 [ 9017.930234] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9017.938175] pending: vmstat_update [ 9017.939888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9017.944865] pending: vmstat_update [ 9017.946263] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9017.952263] pending: vmstat_update [ 9017.958150] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9017.971549] pending: vmstat_update [ 9017.972873] workqueue writeback: flags=0x4a [ 9017.983873] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9017.987955] pending: wb_workfn, wb_workfn [ 9017.991635] workqueue xprtiod: flags=0xa [ 9017.994369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9017.999801] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9018.006969] workqueue ipv6_addrconf: flags=0x40008 [ 9018.013265] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9018.020088] pending: addrconf_verify_work [ 9018.025210] workqueue knbd0-recv: flags=0x1a [ 9018.028934] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9018.033212] in-flight: 647:recv_work [ 9018.035964] workqueue ptlrpc_pinger: flags=0x2 [ 9018.040798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9018.052201] pending: ptlrpc_pinger_main [ptlrpc] [ 9018.054296] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9008s workers=2 idle: 652 [ 9032.057020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9032.060166] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9032.060712] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9032.060767] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9032.060772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9032.060827] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9032.060833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9032.060836] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9032.060849] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9032.060854] RSP: 0018:ffffc38542073af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9032.060858] RAX: 0000000000002305 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9032.060860] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9032.060863] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9032.060864] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9032.060867] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9032.060870] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9032.060873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9032.060875] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9032.060884] Call Trace: [ 9032.060890] [ 9032.060895] ? show_regs.cold.9+0x22/0x2f [ 9032.060911] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9032.060918] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9032.060922] ? watchdog+0x40/0x40 [ 9032.060926] ? hrtimer_interrupt+0x128/0x2c0 [ 9032.060931] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9032.060939] ? apic_timer_interrupt+0xf/0x20 [ 9032.060943] [ 9032.060945] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9032.060950] do_raw_spin_lock+0x177/0x190 [ 9032.060955] _raw_spin_lock+0x21/0x30 [ 9032.060959] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9032.061247] ? do_raw_spin_unlock+0x75/0x190 [ 9032.061254] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9032.061480] ? do_wait_intr+0xf0/0xf0 [ 9032.061489] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9032.061697] mdc_close+0x53b/0xd70 [mdc] [ 9032.061737] lmv_close+0x18d/0x490 [lmv] [ 9032.061766] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9032.061884] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9032.061972] ll_md_close+0x274/0x800 [lustre] [ 9032.062056] ll_file_release+0x128/0xc50 [lustre] [ 9032.062143] __fput+0x10d/0x440 [ 9032.062153] ____fput+0x12/0x20 [ 9032.062157] task_work_run+0xc6/0x110 [ 9032.062167] exit_to_usermode_loop+0x1dd/0x1f0 [ 9032.062175] do_syscall_64+0x3d6/0x440 [ 9032.062180] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9032.062187] RIP: 0033:0x7f1a63983cd8 [ 9032.062195] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9032.062199] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9032.062203] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9032.062206] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9032.062208] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9032.062210] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9032.062213] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9032.086674] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9032.086692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9032.086695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9032.086712] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9032.086716] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9032.086724] RAX: 0000000000003ab3 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9032.086727] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9032.086729] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9032.086734] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9032.086737] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9032.086740] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9032.086744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9032.086746] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9032.086756] Call Trace: [ 9032.086761] [ 9032.086767] ? show_regs.cold.9+0x22/0x2f [ 9032.086784] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9032.086794] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9032.086802] ? watchdog+0x40/0x40 [ 9032.086807] ? hrtimer_interrupt+0x128/0x2c0 [ 9032.086812] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9032.086826] ? apic_timer_interrupt+0xf/0x20 [ 9032.086838] [ 9032.086841] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9032.086853] do_raw_spin_lock+0x177/0x190 [ 9032.086858] _raw_spin_lock+0x21/0x30 [ 9032.086863] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9032.087299] ? do_raw_spin_unlock+0x75/0x190 [ 9032.087305] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9032.087754] ? do_wait_intr+0xf0/0xf0 [ 9032.087762] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9032.088050] mdc_close+0x53b/0xd70 [mdc] [ 9032.088362] lmv_close+0x18d/0x490 [lmv] [ 9032.088403] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9032.101741] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9032.101828] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9032.101835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9032.101838] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9032.101853] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9032.101858] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9032.101862] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9032.101866] RDX: 000000000000568a RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9032.101868] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9032.101871] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9032.101873] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9032.101877] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9032.101880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9032.101883] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9032.101891] Call Trace: [ 9032.101896] [ 9032.101903] ? show_regs.cold.9+0x22/0x2f [ 9032.101914] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9032.101921] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9032.101925] ? watchdog+0x40/0x40 [ 9032.101929] ? hrtimer_interrupt+0x128/0x2c0 [ 9032.101933] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9032.101941] ? apic_timer_interrupt+0xf/0x20 [ 9032.101946] [ 9032.101949] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9032.101954] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9032.101959] do_raw_spin_lock+0x177/0x190 [ 9032.101964] _raw_spin_lock+0x21/0x30 [ 9032.101971] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9032.109048] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9032.109220] ll_md_close+0x274/0x800 [lustre] [ 9032.109329] ll_file_release+0x128/0xc50 [lustre] [ 9032.109437] __fput+0x10d/0x440 [ 9032.109647] ____fput+0x12/0x20 [ 9032.109652] task_work_run+0xc6/0x110 [ 9032.109664] exit_to_usermode_loop+0x1dd/0x1f0 [ 9032.109736] do_syscall_64+0x3d6/0x440 [ 9032.109747] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9032.109762] RIP: 0033:0x7f1a63983cd8 [ 9032.109777] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9032.109782] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9032.109787] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9032.109789] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9032.109791] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9032.109797] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9032.109799] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9032.656923] obd_mod_free+0x46/0x140 [ptlrpc] [ 9032.659303] mdc_commit_open+0x91/0xb0 [mdc] [ 9032.661737] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9032.663668] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9032.666131] after_reply+0xe2c/0x1780 [ptlrpc] [ 9032.668954] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9032.678948] ? __next_timer_interrupt+0x160/0x160 [ 9032.681075] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9032.687691] ? do_wait_intr+0xf0/0xf0 [ 9032.689087] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9032.696038] kthread+0x1d1/0x200 [ 9032.697292] ? set_kthread_struct+0x70/0x70 [ 9032.698999] ret_from_fork+0x1f/0x30 [ 9044.053270] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9044.056636] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9044.075713] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9044.080746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9044.086172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9044.088745] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9044.101314] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9044.105661] RAX: 0000000000007518 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9044.110664] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9044.114497] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9044.118021] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9044.121894] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9044.125259] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9044.129368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9044.131817] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9044.135514] Call Trace: [ 9044.136976] [ 9044.137789] ? show_regs.cold.9+0x22/0x2f [ 9044.139952] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9044.142054] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9044.144089] ? watchdog+0x40/0x40 [ 9044.145720] ? hrtimer_interrupt+0x128/0x2c0 [ 9044.147806] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9044.150003] ? apic_timer_interrupt+0xf/0x20 [ 9044.153972] [ 9044.155049] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9044.157958] do_raw_spin_lock+0x177/0x190 [ 9044.160521] _raw_spin_lock+0x21/0x30 [ 9044.162415] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9044.165279] ? do_raw_spin_unlock+0x75/0x190 [ 9044.167030] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9044.169633] ? do_wait_intr+0xf0/0xf0 [ 9044.171341] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9044.174338] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9044.176200] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9044.178897] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9044.181842] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9044.184834] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9044.186881] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9044.188282] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9044.191154] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9044.193647] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9044.196861] lmv_intent_open+0x338/0x1150 [lmv] [ 9044.198503] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9044.202776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9044.205861] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9044.210164] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9044.213966] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9044.219389] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9044.221345] ? do_raw_spin_unlock+0x75/0x190 [ 9044.223976] ? _raw_spin_unlock+0x12/0x30 [ 9044.225371] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9044.229025] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9044.231710] ll_atomic_open+0x36a/0xd70 [lustre] [ 9044.233733] lookup_open+0x356/0xb40 [ 9044.235446] do_last+0x313/0xfc0 [ 9044.236802] ? nd_jump_root+0xe5/0x160 [ 9044.238544] ? path_init+0x437/0x520 [ 9044.240109] path_openat+0xf7/0x500 [ 9044.241659] ? do_raw_spin_unlock+0x75/0x190 [ 9044.243531] do_filp_open+0x99/0x140 [ 9044.245157] ? getname_flags+0x6e/0x330 [ 9044.246951] ? __check_object_size+0xff/0x256 [ 9044.249075] ? do_raw_spin_unlock+0x75/0x190 [ 9044.250947] ? _raw_spin_unlock+0x12/0x30 [ 9044.252481] do_sys_openat2+0x2b4/0x410 [ 9044.253826] do_sys_open+0x73/0xa0 [ 9044.255511] __x64_sys_openat+0x24/0x30 [ 9044.257094] do_syscall_64+0xc1/0x440 [ 9044.258702] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9044.260795] RIP: 0033:0x7f1a63983332 [ 9044.262492] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9044.270568] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9044.273773] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9044.276848] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9044.279833] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9044.282691] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9044.285584] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9048.545188] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7025s! [ 9048.551930] Showing busy workqueues and worker pools: [ 9048.558033] workqueue events: flags=0x0 [ 9048.560749] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9048.563739] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9048.567367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9048.571038] pending: sec_gc_main [ptlrpc] [ 9048.573719] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9048.576551] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9048.581761] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9048.584747] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9048.589678] workqueue events_long: flags=0x0 [ 9048.591467] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9048.593947] pending: lu_objects_flush [obdclass] [ 9048.596562] workqueue events_unbound: flags=0x2 [ 9048.598266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9048.600684] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9048.605263] workqueue events_power_efficient: flags=0x80 [ 9048.607108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9048.609490] pending: neigh_periodic_work, do_cache_clean [ 9048.611498] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9048.614086] pending: neigh_periodic_work, check_lifetime [ 9048.616320] workqueue mm_percpu_wq: flags=0x8 [ 9048.618934] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9048.621631] pending: vmstat_update [ 9048.623643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9048.627238] pending: vmstat_update [ 9048.629393] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9048.631908] pending: vmstat_update [ 9048.633328] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9048.636282] pending: vmstat_update [ 9048.638548] workqueue writeback: flags=0x4a [ 9048.640759] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9048.643939] pending: wb_workfn, wb_workfn [ 9048.645924] workqueue xprtiod: flags=0xa [ 9048.647769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9048.650119] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9048.653267] workqueue ipv6_addrconf: flags=0x40008 [ 9048.654768] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9048.656801] pending: addrconf_verify_work [ 9048.658234] workqueue knbd0-recv: flags=0x1a [ 9048.659708] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9048.661730] in-flight: 647:recv_work [ 9048.662944] workqueue ptlrpc_pinger: flags=0x2 [ 9048.664679] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9048.667524] pending: ptlrpc_pinger_main [ptlrpc] [ 9048.670819] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9038s workers=2 idle: 652 [ 9060.056028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9060.057345] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9060.061539] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9060.061553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9060.061632] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9060.061639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9060.061642] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9060.061656] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9060.061660] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9060.061665] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9060.061667] RDX: 00000000000013e3 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9060.061669] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9060.061672] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9060.061674] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9060.061678] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9060.061681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9060.061683] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9060.061693] Call Trace: [ 9060.061699] [ 9060.061704] ? show_regs.cold.9+0x22/0x2f [ 9060.061715] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9060.061722] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9060.061778] Modules linked in: lustre(O) osp(O) [ 9060.062663] ? watchdog+0x40/0x40 [ 9060.062670] ? hrtimer_interrupt+0x128/0x2c0 [ 9060.062675] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9060.062683] ? apic_timer_interrupt+0xf/0x20 [ 9060.062688] [ 9060.062691] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9060.062696] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9060.062701] do_raw_spin_lock+0x177/0x190 [ 9060.062707] _raw_spin_lock+0x21/0x30 [ 9060.062713] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9060.064953] Modules linked in: [ 9060.063618] obd_mod_free+0x46/0x140 [ptlrpc] [ 9060.065117] mdc_commit_open+0x91/0xb0 [mdc] [ 9060.065150] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9060.065370] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9060.065604] after_reply+0xe2c/0x1780 [ptlrpc] [ 9060.069420] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9060.069648] ? __next_timer_interrupt+0x160/0x160 [ 9060.069658] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9060.069905] ? do_wait_intr+0xf0/0xf0 [ 9060.069915] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9060.070128] kthread+0x1d1/0x200 [ 9060.070134] ? set_kthread_struct+0x70/0x70 [ 9060.070138] ret_from_fork+0x1f/0x30 [ 9060.072326] ofd(O) [ 9060.096193] lustre(O) [ 9060.103785] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9060.103863] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9060.103874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9060.103877] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9060.103891] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9060.103896] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9060.103901] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9060.103904] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9060.103906] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9060.103908] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9060.103911] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9060.103914] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9060.103917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9060.103919] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9060.103930] Call Trace: [ 9060.103947] [ 9060.103954] ? show_regs.cold.9+0x22/0x2f [ 9060.103964] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9060.103970] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9060.103974] ? watchdog+0x40/0x40 [ 9060.103979] ? hrtimer_interrupt+0x128/0x2c0 [ 9060.103986] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9060.103995] ? apic_timer_interrupt+0xf/0x20 [ 9060.103999] [ 9060.104003] ? native_safe_halt+0x2e/0x50 [ 9060.104007] kvm_wait+0xcd/0xe0 [ 9060.104014] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9060.104022] do_raw_spin_lock+0x177/0x190 [ 9060.104027] _raw_spin_lock+0x21/0x30 [ 9060.104035] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9060.104335] ? do_raw_spin_unlock+0x75/0x190 [ 9060.104341] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9060.104553] ? do_wait_intr+0xf0/0xf0 [ 9060.104560] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9060.104875] mdc_close+0x53b/0xd70 [mdc] [ 9060.115192] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9060.115260] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9060.115268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9060.115271] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9060.115284] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9060.115287] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9060.115291] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9060.115293] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9060.115295] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9060.115297] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9060.115299] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9060.115302] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9060.115305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9060.115307] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9060.115315] Call Trace: [ 9060.115321] [ 9060.115328] ? show_regs.cold.9+0x22/0x2f [ 9060.115337] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9060.115344] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9060.115348] ? watchdog+0x40/0x40 [ 9060.115353] ? hrtimer_interrupt+0x128/0x2c0 [ 9060.115357] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9060.115364] ? apic_timer_interrupt+0xf/0x20 [ 9060.115369] [ 9060.115371] ? native_safe_halt+0x2e/0x50 [ 9060.115376] kvm_wait+0xcd/0xe0 [ 9060.121480] lmv_close+0x18d/0x490 [lmv] [ 9060.152431] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9060.152457] do_raw_spin_lock+0x177/0x190 [ 9060.152463] _raw_spin_lock+0x21/0x30 [ 9060.152469] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9060.152748] ? do_raw_spin_unlock+0x75/0x190 [ 9060.152754] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9060.152950] ? do_wait_intr+0xf0/0xf0 [ 9060.152957] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9060.156677] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9060.156793] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9060.160225] mdc_close+0x53b/0xd70 [mdc] [ 9060.164073] ll_md_close+0x274/0x800 [lustre] [ 9060.168419] lmv_close+0x18d/0x490 [lmv] [ 9060.172945] ll_file_release+0x128/0xc50 [lustre] [ 9060.173076] __fput+0x10d/0x440 [ 9060.173086] ____fput+0x12/0x20 [ 9060.173090] task_work_run+0xc6/0x110 [ 9060.173098] exit_to_usermode_loop+0x1dd/0x1f0 [ 9060.173104] do_syscall_64+0x3d6/0x440 [ 9060.173109] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9060.173117] RIP: 0033:0x7f1a63983cd8 [ 9060.173125] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9060.173130] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9060.173135] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9060.173137] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9060.173139] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9060.173141] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9060.173143] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9060.672466] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9060.676544] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9060.679332] ll_md_close+0x274/0x800 [lustre] [ 9060.681480] ll_file_release+0x128/0xc50 [lustre] [ 9060.684816] __fput+0x10d/0x440 [ 9060.686597] ____fput+0x12/0x20 [ 9060.688372] task_work_run+0xc6/0x110 [ 9060.689966] exit_to_usermode_loop+0x1dd/0x1f0 [ 9060.692235] do_syscall_64+0x3d6/0x440 [ 9060.694196] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9060.698151] RIP: 0033:0x7f1a63983cd8 [ 9060.700427] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9060.720282] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9060.727139] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9060.730989] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9060.742925] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9060.747484] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9060.754926] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9076.054183] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9076.066386] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9076.096990] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9076.103882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9076.113078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9076.118660] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9076.133379] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9076.138456] RAX: 0000000000002816 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9076.144006] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9076.149209] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9076.153870] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9076.159532] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9076.165441] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9076.171977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9076.176304] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9076.181252] Call Trace: [ 9076.183009] [ 9076.184188] ? show_regs.cold.9+0x22/0x2f [ 9076.186689] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9076.189457] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9076.192545] ? watchdog+0x40/0x40 [ 9076.197819] ? hrtimer_interrupt+0x128/0x2c0 [ 9076.201727] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9076.206370] ? apic_timer_interrupt+0xf/0x20 [ 9076.208841] [ 9076.210354] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9076.213421] do_raw_spin_lock+0x177/0x190 [ 9076.215102] _raw_spin_lock+0x21/0x30 [ 9076.217010] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9076.221153] ? do_raw_spin_unlock+0x75/0x190 [ 9076.224264] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9076.226958] ? do_wait_intr+0xf0/0xf0 [ 9076.228761] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9076.231023] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9076.233419] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9076.236257] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.238510] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9076.240820] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9076.242942] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9076.245158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.247804] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9076.250703] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9076.254745] lmv_intent_open+0x338/0x1150 [lmv] [ 9076.256740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.259591] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.262285] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9076.264817] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9076.271248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.274111] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9076.276722] ? do_raw_spin_unlock+0x75/0x190 [ 9076.279726] ? _raw_spin_unlock+0x12/0x30 [ 9076.282316] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9076.285683] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9076.288566] ll_atomic_open+0x36a/0xd70 [lustre] [ 9076.291483] lookup_open+0x356/0xb40 [ 9076.293567] do_last+0x313/0xfc0 [ 9076.295496] ? nd_jump_root+0xe5/0x160 [ 9076.297689] ? path_init+0x437/0x520 [ 9076.299749] path_openat+0xf7/0x500 [ 9076.302746] ? do_raw_spin_unlock+0x75/0x190 [ 9076.305182] do_filp_open+0x99/0x140 [ 9076.307739] ? getname_flags+0x6e/0x330 [ 9076.310054] ? __check_object_size+0xff/0x256 [ 9076.312532] ? do_raw_spin_unlock+0x75/0x190 [ 9076.314774] ? _raw_spin_unlock+0x12/0x30 [ 9076.322221] do_sys_openat2+0x2b4/0x410 [ 9076.326828] do_sys_open+0x73/0xa0 [ 9076.331427] __x64_sys_openat+0x24/0x30 [ 9076.334745] do_syscall_64+0xc1/0x440 [ 9076.340420] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9076.345804] RIP: 0033:0x7f1a63983332 [ 9076.350714] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9076.370331] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9076.376003] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9076.380089] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9076.393697] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9076.401588] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9076.409726] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9078.753969] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7055s! [ 9078.761419] Showing busy workqueues and worker pools: [ 9078.767266] workqueue events: flags=0x0 [ 9078.777198] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9078.790044] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9078.794692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9078.800294] pending: sec_gc_main [ptlrpc] [ 9078.802996] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9078.807822] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9078.813157] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9078.822074] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9078.842428] workqueue events_long: flags=0x0 [ 9078.847739] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9078.855595] pending: lu_objects_flush [obdclass] [ 9078.859660] workqueue events_unbound: flags=0x2 [ 9078.863107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9078.869746] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9078.878129] workqueue events_power_efficient: flags=0x80 [ 9078.881050] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9078.884196] pending: neigh_periodic_work, do_cache_clean [ 9078.887812] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9078.894715] pending: neigh_periodic_work, check_lifetime [ 9078.900553] workqueue mm_percpu_wq: flags=0x8 [ 9078.903050] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9078.907073] pending: vmstat_update [ 9078.911616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9078.915585] pending: vmstat_update [ 9078.919185] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9078.926443] pending: vmstat_update [ 9078.928617] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9078.933176] pending: vmstat_update [ 9078.934659] workqueue writeback: flags=0x4a [ 9078.937555] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9078.941177] pending: wb_workfn, wb_workfn [ 9078.943369] workqueue xprtiod: flags=0xa [ 9078.945221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9078.952740] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9078.956139] workqueue ipv6_addrconf: flags=0x40008 [ 9078.957310] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9078.963384] pending: addrconf_verify_work [ 9078.964597] workqueue knbd0-recv: flags=0x1a [ 9078.967044] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9078.972325] in-flight: 647:recv_work [ 9078.974920] workqueue ptlrpc_pinger: flags=0x2 [ 9078.977048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9078.980534] pending: ptlrpc_pinger_main [ptlrpc] [ 9078.983709] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9069s workers=2 idle: 652 [ 9088.056081] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9088.058064] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9088.059195] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9088.059264] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9088.059271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9088.059273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9088.059285] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9088.059289] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9088.059293] RAX: 0000000000006b43 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9088.059295] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9088.059297] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9088.059299] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9088.059301] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9088.059304] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9088.059307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9088.059309] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9088.059318] Call Trace: [ 9088.059323] [ 9088.059329] ? show_regs.cold.9+0x22/0x2f [ 9088.059339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9088.059346] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9088.059350] ? watchdog+0x40/0x40 [ 9088.059354] ? hrtimer_interrupt+0x128/0x2c0 [ 9088.059358] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9088.059366] ? apic_timer_interrupt+0xf/0x20 [ 9088.059371] [ 9088.059375] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9088.059382] do_raw_spin_lock+0x177/0x190 [ 9088.059534] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9088.059539] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9088.059577] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9088.059580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9088.059581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9088.059587] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9088.059590] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9088.059592] RAX: 0000000000003514 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9088.059594] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9088.059596] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9088.059597] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9088.059599] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9088.059601] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9088.059603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9088.059604] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9088.059610] Call Trace: [ 9088.059611] [ 9088.059614] ? show_regs.cold.9+0x22/0x2f [ 9088.059619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9088.059623] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9088.059625] ? watchdog+0x40/0x40 [ 9088.059628] ? hrtimer_interrupt+0x128/0x2c0 [ 9088.059630] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9088.059635] ? apic_timer_interrupt+0xf/0x20 [ 9088.059638] [ 9088.059640] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9088.059643] do_raw_spin_lock+0x177/0x190 [ 9088.059647] _raw_spin_lock+0x21/0x30 [ 9088.059650] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9088.060347] ? do_raw_spin_unlock+0x75/0x190 [ 9088.060352] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9088.061625] ? do_wait_intr+0xf0/0xf0 [ 9088.061636] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9088.061998] mdc_close+0x53b/0xd70 [mdc] [ 9088.062027] lmv_close+0x18d/0x490 [lmv] [ 9088.062046] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9088.062123] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9088.062240] ll_md_close+0x274/0x800 [lustre] [ 9088.062311] ll_file_release+0x128/0xc50 [lustre] [ 9088.062363] __fput+0x10d/0x440 [ 9088.062371] ____fput+0x12/0x20 [ 9088.062374] task_work_run+0xc6/0x110 [ 9088.062379] exit_to_usermode_loop+0x1dd/0x1f0 [ 9088.062385] do_syscall_64+0x3d6/0x440 [ 9088.062388] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9088.062394] RIP: 0033:0x7f1a63983cd8 [ 9088.062399] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9088.062402] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9088.062406] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9088.062407] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9088.062408] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9088.062535] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9088.062536] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9088.063162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9088.063292] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9088.063298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9088.063301] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9088.063310] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9088.063314] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9088.063319] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9088.063322] RDX: 000000000000654b RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9088.063325] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9088.063327] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9088.063329] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9088.063332] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9088.063335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9088.063338] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9088.063347] Call Trace: [ 9088.063351] [ 9088.063355] ? show_regs.cold.9+0x22/0x2f [ 9088.063364] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9088.063370] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9088.063374] ? watchdog+0x40/0x40 [ 9088.063378] ? hrtimer_interrupt+0x128/0x2c0 [ 9088.063382] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9088.063398] ? apic_timer_interrupt+0xf/0x20 [ 9088.063403] [ 9088.063406] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9088.063410] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9088.063414] do_raw_spin_lock+0x177/0x190 [ 9088.063419] _raw_spin_lock+0x21/0x30 [ 9088.063424] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9088.063740] obd_mod_free+0x46/0x140 [ptlrpc] [ 9088.064137] mdc_commit_open+0x91/0xb0 [mdc] [ 9088.064181] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9088.064419] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9088.064766] after_reply+0xe2c/0x1780 [ptlrpc] [ 9088.065001] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9088.065194] ? __next_timer_interrupt+0x160/0x160 [ 9088.065209] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9088.065425] ? do_wait_intr+0xf0/0xf0 [ 9088.065435] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9088.065911] kthread+0x1d1/0x200 [ 9088.065917] ? set_kthread_struct+0x70/0x70 [ 9088.065921] ret_from_fork+0x1f/0x30 [ 9089.141267] _raw_spin_lock+0x21/0x30 [ 9089.143109] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9089.145567] ? do_raw_spin_unlock+0x75/0x190 [ 9089.147569] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9089.150586] ? do_wait_intr+0xf0/0xf0 [ 9089.152280] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9089.155181] mdc_close+0x53b/0xd70 [mdc] [ 9089.161515] lmv_close+0x18d/0x490 [lmv] [ 9089.163509] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9089.169161] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9089.172323] ll_md_close+0x274/0x800 [lustre] [ 9089.176016] ll_file_release+0x128/0xc50 [lustre] [ 9089.179062] __fput+0x10d/0x440 [ 9089.182086] ____fput+0x12/0x20 [ 9089.185038] task_work_run+0xc6/0x110 [ 9089.189827] exit_to_usermode_loop+0x1dd/0x1f0 [ 9089.193528] do_syscall_64+0x3d6/0x440 [ 9089.195586] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9089.197456] RIP: 0033:0x7f1a63983cd8 [ 9089.199186] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9089.208981] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9089.222479] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9089.232540] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9089.237500] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9089.246380] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9089.250439] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9102.595342] rcu: INFO: rcu_sched self-detected stall on CPU [ 9102.599591] rcu: 3-...!: (4468812 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 9102.618180] (t=7080211 jiffies g=269633 q=2528) [ 9102.627318] rcu: rcu_sched kthread starved for 7079861 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 9102.641600] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9102.647797] rcu: RCU grace-period kthread stack dump: [ 9102.650715] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9102.664242] Call Trace: [ 9102.670117] __schedule+0x351/0xcb0 [ 9102.674380] schedule+0xc0/0x180 [ 9102.677609] schedule_timeout+0xb4/0x190 [ 9102.681285] ? do_raw_spin_unlock+0x75/0x190 [ 9102.695249] ? __next_timer_interrupt+0x160/0x160 [ 9102.703248] ? prepare_to_swait_event+0xf3/0x140 [ 9102.707115] rcu_gp_kthread+0x8dd/0x15b0 [ 9102.711925] ? force_qs_rnp+0x2c0/0x2c0 [ 9102.714177] kthread+0x1d1/0x200 [ 9102.715751] ? set_kthread_struct+0x70/0x70 [ 9102.721243] ret_from_fork+0x1f/0x30 [ 9102.727738] rcu: Stack dump where RCU GP kthread last ran: [ 9102.737778] Sending NMI from CPU 3 to CPUs 0: [ 9102.745235] NMI backtrace for cpu 0 [ 9102.745242] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9102.745250] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9102.745251] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9102.745253] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9102.745255] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 [ 9102.745258] RAX: 000000000000626d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9102.745260] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9102.745261] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9102.745263] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9102.745264] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9102.745266] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9102.745267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9102.745268] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9102.745270] Call Trace: [ 9102.745270] [ 9102.745271] ? show_regs.cold.9+0x22/0x2f [ 9102.745272] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9102.745274] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9102.745275] ? nmi_handle+0xbd/0x210 [ 9102.745276] ? default_do_nmi+0x4b/0x1b0 [ 9102.745277] ? do_nmi+0x280/0x380 [ 9102.745278] ? end_repeat_nmi+0x16/0x1f [ 9102.745279] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9102.745281] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9102.745282] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9102.745283] [ 9102.745284] do_raw_spin_lock+0x177/0x190 [ 9102.745285] _raw_spin_lock+0x21/0x30 [ 9102.745287] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9102.745291] ? do_raw_spin_unlock+0x75/0x190 [ 9102.745293] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9102.745294] ? do_wait_intr+0xf0/0xf0 [ 9102.745295] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9102.745296] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9102.745300] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9102.745301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9102.745303] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9102.745304] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9102.745305] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9102.745306] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9102.745307] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9102.745308] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9102.745309] lmv_intent_open+0x338/0x1150 [lmv] [ 9102.745311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9102.745311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9102.745313] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9102.745314] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9102.745315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9102.745316] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9102.745318] ? do_raw_spin_unlock+0x75/0x190 [ 9102.745319] ? _raw_spin_unlock+0x12/0x30 [ 9102.745320] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9102.745321] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9102.745322] ll_atomic_open+0x36a/0xd70 [lustre] [ 9102.745323] lookup_open+0x356/0xb40 [ 9102.745324] do_last+0x313/0xfc0 [ 9102.745325] ? nd_jump_root+0xe5/0x160 [ 9102.745326] ? path_init+0x437/0x520 [ 9102.745327] path_openat+0xf7/0x500 [ 9102.745328] ? do_raw_spin_unlock+0x75/0x190 [ 9102.745329] do_filp_open+0x99/0x140 [ 9102.745330] ? getname_flags+0x6e/0x330 [ 9102.745331] ? __check_object_size+0xff/0x256 [ 9102.745332] ? do_raw_spin_unlock+0x75/0x190 [ 9102.745333] ? _raw_spin_unlock+0x12/0x30 [ 9102.745334] do_sys_openat2+0x2b4/0x410 [ 9102.745334] do_sys_open+0x73/0xa0 [ 9102.745335] __x64_sys_openat+0x24/0x30 [ 9102.745336] do_syscall_64+0xc1/0x440 [ 9102.745337] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9102.745338] RIP: 0033:0x7f1a63983332 [ 9102.745339] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9102.745341] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9102.745343] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9102.745344] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9102.745345] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9102.745347] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9102.745348] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9102.745669] NMI backtrace for cpu 3 [ 9102.996944] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9103.004036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9103.009316] Call Trace: [ 9103.010336] [ 9103.011217] ? dump_stack+0xbb/0x10e [ 9103.012755] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9103.014722] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9103.016743] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9103.019281] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9103.021477] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9103.024033] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9103.028867] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9103.032929] ? update_process_times+0x6d/0xc0 [ 9103.035360] ? tick_sched_handle.isra.18+0x27/0x90 [ 9103.037830] ? tick_sched_timer+0x53/0xb0 [ 9103.039611] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9103.041676] ? tick_sched_handle.isra.18+0x90/0x90 [ 9103.043735] ? hrtimer_interrupt+0x128/0x2c0 [ 9103.045607] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9103.048036] ? apic_timer_interrupt+0xf/0x20 [ 9103.059465] [ 9103.061246] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9103.065293] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9103.070232] ? do_raw_spin_lock+0x177/0x190 [ 9103.076544] ? _raw_spin_lock+0x21/0x30 [ 9103.080735] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9103.082447] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9103.089272] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9103.091543] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9103.098678] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9103.107256] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9103.116532] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9103.128339] ? __next_timer_interrupt+0x160/0x160 [ 9103.137594] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9103.144886] ? do_wait_intr+0xf0/0xf0 [ 9103.151372] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9103.160776] ? kthread+0x1d1/0x200 [ 9103.162677] ? set_kthread_struct+0x70/0x70 [ 9103.165315] ? ret_from_fork+0x1f/0x30 [ 9109.472268] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7086s! [ 9109.478095] Showing busy workqueues and worker pools: [ 9109.489248] workqueue events: flags=0x0 [ 9109.497042] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9109.499845] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9109.503378] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9109.506472] pending: sec_gc_main [ptlrpc] [ 9109.509185] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9109.512330] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9109.518979] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9109.522091] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9109.528149] workqueue events_long: flags=0x0 [ 9109.530227] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9109.534299] pending: lu_objects_flush [obdclass] [ 9109.539827] workqueue events_unbound: flags=0x2 [ 9109.544340] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9109.555876] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9109.567951] workqueue events_power_efficient: flags=0x80 [ 9109.570847] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9109.575793] pending: neigh_periodic_work, do_cache_clean [ 9109.579294] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9109.583119] pending: neigh_periodic_work, check_lifetime [ 9109.587890] workqueue mm_percpu_wq: flags=0x8 [ 9109.590055] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9109.593209] pending: vmstat_update [ 9109.594950] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9109.598207] pending: vmstat_update [ 9109.599909] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9109.607119] pending: vmstat_update [ 9109.608507] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9109.611840] pending: vmstat_update [ 9109.613114] workqueue writeback: flags=0x4a [ 9109.614523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9109.625060] pending: wb_workfn, wb_workfn [ 9109.636909] workqueue xprtiod: flags=0xa [ 9109.638494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9109.649357] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9109.659100] workqueue ipv6_addrconf: flags=0x40008 [ 9109.664584] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9109.675875] pending: addrconf_verify_work [ 9109.678973] workqueue knbd0-recv: flags=0x1a [ 9109.681241] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9109.687653] in-flight: 647:recv_work [ 9109.689259] workqueue ptlrpc_pinger: flags=0x2 [ 9109.692012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9109.695297] pending: ptlrpc_pinger_main [ptlrpc] [ 9109.699188] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9099s workers=2 idle: 652 [ 9116.058030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9116.063050] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9116.065148] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9116.065245] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9116.065252] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9116.065254] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9116.065269] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9116.065273] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9116.065278] RAX: 00000000000079c2 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9116.065280] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9116.065283] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9116.065284] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9116.065287] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9116.065290] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9116.065293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9116.065296] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9116.065483] Call Trace: [ 9116.065556] [ 9116.065563] ? show_regs.cold.9+0x22/0x2f [ 9116.065573] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9116.065580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9116.065584] ? watchdog+0x40/0x40 [ 9116.065588] ? hrtimer_interrupt+0x128/0x2c0 [ 9116.065592] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9116.065600] ? apic_timer_interrupt+0xf/0x20 [ 9116.065606] [ 9116.065609] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9116.065615] do_raw_spin_lock+0x177/0x190 [ 9116.065619] _raw_spin_lock+0x21/0x30 [ 9116.065623] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9116.066027] ? do_raw_spin_unlock+0x75/0x190 [ 9116.066034] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9116.066307] ? do_wait_intr+0xf0/0xf0 [ 9116.066315] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9116.075069] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9116.075162] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9116.075171] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9116.075174] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9116.075194] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9116.075199] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9116.075203] RAX: 0000000000007e85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9116.075206] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9116.075208] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9116.075210] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9116.075213] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9116.075216] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9116.075219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9116.075221] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9116.075229] Call Trace: [ 9116.075236] [ 9116.075243] ? show_regs.cold.9+0x22/0x2f [ 9116.075263] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9116.075270] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9116.075276] ? watchdog+0x40/0x40 [ 9116.075280] ? hrtimer_interrupt+0x128/0x2c0 [ 9116.075284] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9116.075294] ? apic_timer_interrupt+0xf/0x20 [ 9116.075298] [ 9116.075301] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9116.075307] do_raw_spin_lock+0x177/0x190 [ 9116.075312] _raw_spin_lock+0x21/0x30 [ 9116.075317] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9116.075643] ? do_raw_spin_unlock+0x75/0x190 [ 9116.075649] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9116.075822] ? do_wait_intr+0xf0/0xf0 [ 9116.075829] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9116.075985] mdc_close+0x53b/0xd70 [mdc] [ 9116.076034] lmv_close+0x18d/0x490 [lmv] [ 9116.076070] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9116.076195] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9116.076275] ll_md_close+0x274/0x800 [lustre] [ 9116.076356] ll_file_release+0x128/0xc50 [lustre] [ 9116.076438] __fput+0x10d/0x440 [ 9116.076448] ____fput+0x12/0x20 [ 9116.076453] task_work_run+0xc6/0x110 [ 9116.076462] exit_to_usermode_loop+0x1dd/0x1f0 [ 9116.076471] do_syscall_64+0x3d6/0x440 [ 9116.076476] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9116.076483] RIP: 0033:0x7f1a63983cd8 [ 9116.076492] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9116.076496] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9116.076501] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9116.076514] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9116.076517] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9116.076519] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9116.076521] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9116.543546] mdc_close+0x53b/0xd70 [mdc] [ 9116.551763] lmv_close+0x18d/0x490 [lmv] [ 9116.555666] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9116.559324] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9116.562200] ll_md_close+0x274/0x800 [lustre] [ 9116.564851] ll_file_release+0x128/0xc50 [lustre] [ 9116.566918] __fput+0x10d/0x440 [ 9116.568544] ____fput+0x12/0x20 [ 9116.570286] task_work_run+0xc6/0x110 [ 9116.572225] exit_to_usermode_loop+0x1dd/0x1f0 [ 9116.575064] do_syscall_64+0x3d6/0x440 [ 9116.577842] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9116.584015] RIP: 0033:0x7f1a63983cd8 [ 9116.589997] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9116.600928] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9116.605413] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9116.609357] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9116.613310] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9116.616732] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9116.620380] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9128.058324] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 9128.063257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9128.103742] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9128.109817] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9128.116758] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9128.124209] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9128.145751] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9128.154479] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9128.163093] RDX: 000000000000381e RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9128.167567] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9128.170296] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9128.173667] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9128.177024] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9128.180538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9128.182855] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9128.184999] Call Trace: [ 9128.185824] [ 9128.186292] ? show_regs.cold.9+0x22/0x2f [ 9128.187992] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9128.189467] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9128.192771] ? watchdog+0x40/0x40 [ 9128.193612] ? hrtimer_interrupt+0x128/0x2c0 [ 9128.194886] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9128.197217] ? apic_timer_interrupt+0xf/0x20 [ 9128.199110] [ 9128.200535] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9128.203092] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9128.204890] do_raw_spin_lock+0x177/0x190 [ 9128.206804] _raw_spin_lock+0x21/0x30 [ 9128.208592] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9128.211567] obd_mod_free+0x46/0x140 [ptlrpc] [ 9128.214689] mdc_commit_open+0x91/0xb0 [mdc] [ 9128.216565] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9128.219206] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9128.223185] after_reply+0xe2c/0x1780 [ptlrpc] [ 9128.225915] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9128.228952] ? __next_timer_interrupt+0x160/0x160 [ 9128.231982] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9128.236895] ? do_wait_intr+0xf0/0xf0 [ 9128.238203] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9128.243898] kthread+0x1d1/0x200 [ 9128.245615] ? set_kthread_struct+0x70/0x70 [ 9128.249733] ret_from_fork+0x1f/0x30 [ 9136.053012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9136.057230] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9136.075924] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9136.080098] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9136.083622] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9136.086033] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9136.093756] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9136.097112] RAX: 0000000000007acb RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9136.099661] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9136.102255] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9136.104646] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9136.106954] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9136.109530] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9136.112553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9136.114416] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9136.116694] Call Trace: [ 9136.117547] [ 9136.118234] ? show_regs.cold.9+0x22/0x2f [ 9136.119454] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9136.121270] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9136.122848] ? watchdog+0x40/0x40 [ 9136.123869] ? hrtimer_interrupt+0x128/0x2c0 [ 9136.125172] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9136.126725] ? apic_timer_interrupt+0xf/0x20 [ 9136.128058] [ 9136.128909] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9136.130912] do_raw_spin_lock+0x177/0x190 [ 9136.132138] _raw_spin_lock+0x21/0x30 [ 9136.133503] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9136.135862] ? do_raw_spin_unlock+0x75/0x190 [ 9136.137105] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9136.139334] ? do_wait_intr+0xf0/0xf0 [ 9136.140658] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9136.142861] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9136.144893] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9136.147673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9136.149545] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9136.151294] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9136.153189] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9136.154586] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9136.156609] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9136.159127] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9136.161232] lmv_intent_open+0x338/0x1150 [lmv] [ 9136.162759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9136.164629] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9136.166554] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9136.168170] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9136.169651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9136.171535] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9136.173458] ? do_raw_spin_unlock+0x75/0x190 [ 9136.174746] ? _raw_spin_unlock+0x12/0x30 [ 9136.176249] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9136.177957] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9136.179791] ll_atomic_open+0x36a/0xd70 [lustre] [ 9136.181463] lookup_open+0x356/0xb40 [ 9136.182598] do_last+0x313/0xfc0 [ 9136.183627] ? nd_jump_root+0xe5/0x160 [ 9136.184985] ? path_init+0x437/0x520 [ 9136.186219] path_openat+0xf7/0x500 [ 9136.187727] ? do_raw_spin_unlock+0x75/0x190 [ 9136.189463] do_filp_open+0x99/0x140 [ 9136.190890] ? getname_flags+0x6e/0x330 [ 9136.192365] ? __check_object_size+0xff/0x256 [ 9136.193986] ? do_raw_spin_unlock+0x75/0x190 [ 9136.195733] ? _raw_spin_unlock+0x12/0x30 [ 9136.197096] do_sys_openat2+0x2b4/0x410 [ 9136.198740] do_sys_open+0x73/0xa0 [ 9136.199959] __x64_sys_openat+0x24/0x30 [ 9136.201363] do_syscall_64+0xc1/0x440 [ 9136.202655] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9136.204588] RIP: 0033:0x7f1a63983332 [ 9136.205852] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9136.212888] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9136.215744] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9136.218574] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9136.221512] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9136.224193] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9136.227733] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9140.200307] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7117s! [ 9140.214167] Showing busy workqueues and worker pools: [ 9140.217175] workqueue events: flags=0x0 [ 9140.219492] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9140.226442] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9140.229609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9140.232990] pending: sec_gc_main [ptlrpc] [ 9140.240088] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9140.247960] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9140.262453] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9140.271928] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9140.282829] workqueue events_long: flags=0x0 [ 9140.284743] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9140.293685] pending: lu_objects_flush [obdclass] [ 9140.295558] workqueue events_unbound: flags=0x2 [ 9140.300074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9140.304002] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9140.309573] workqueue events_power_efficient: flags=0x80 [ 9140.313851] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9140.318596] pending: neigh_periodic_work, do_cache_clean [ 9140.323078] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9140.329569] pending: neigh_periodic_work, check_lifetime [ 9140.336221] workqueue mm_percpu_wq: flags=0x8 [ 9140.342488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9140.348636] pending: vmstat_update [ 9140.352559] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9140.359428] pending: vmstat_update [ 9140.363746] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9140.373471] pending: vmstat_update [ 9140.376461] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9140.379455] pending: vmstat_update [ 9140.390633] workqueue writeback: flags=0x4a [ 9140.394490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9140.403520] pending: wb_workfn, wb_workfn [ 9140.409801] workqueue xprtiod: flags=0xa [ 9140.417818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9140.423490] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9140.430443] workqueue ipv6_addrconf: flags=0x40008 [ 9140.436772] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9140.444225] pending: addrconf_verify_work [ 9140.450132] workqueue knbd0-recv: flags=0x1a [ 9140.451784] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9140.461221] in-flight: 647:recv_work [ 9140.465745] workqueue ptlrpc_pinger: flags=0x2 [ 9140.474906] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9140.482810] pending: ptlrpc_pinger_main [ptlrpc] [ 9140.490039] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9130s workers=2 idle: 652 [ 9144.060347] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 9144.065804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9144.068346] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 9144.104618] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9144.104632] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9144.104635] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9144.104650] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9144.104654] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9144.104659] RAX: 0000000000001a6a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9144.104662] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9144.104664] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9144.104667] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9144.104669] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9144.104672] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9144.104675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9144.104678] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9144.104687] Call Trace: [ 9144.104692] [ 9144.104698] ? show_regs.cold.9+0x22/0x2f [ 9144.104708] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9144.104714] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9144.104719] ? watchdog+0x40/0x40 [ 9144.104723] ? hrtimer_interrupt+0x128/0x2c0 [ 9144.104728] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9144.104738] ? apic_timer_interrupt+0xf/0x20 [ 9144.104742] [ 9144.104745] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9144.104752] do_raw_spin_lock+0x177/0x190 [ 9144.104756] _raw_spin_lock+0x21/0x30 [ 9144.104760] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9144.105040] ? do_raw_spin_unlock+0x75/0x190 [ 9144.105046] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9144.105555] ? do_wait_intr+0xf0/0xf0 [ 9144.105564] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9144.127052] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9144.127132] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9144.127140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9144.127143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9144.127160] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9144.127165] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9144.127169] RAX: 000000000000020e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9144.127172] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9144.127175] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9144.127177] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9144.127179] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9144.127183] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9144.127186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9144.127189] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9144.127199] Call Trace: [ 9144.127204] [ 9144.127210] ? show_regs.cold.9+0x22/0x2f [ 9144.127220] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9144.127226] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9144.127230] ? watchdog+0x40/0x40 [ 9144.127246] ? hrtimer_interrupt+0x128/0x2c0 [ 9144.127249] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9144.127256] ? apic_timer_interrupt+0xf/0x20 [ 9144.127261] [ 9144.127264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9144.127270] do_raw_spin_lock+0x177/0x190 [ 9144.127275] _raw_spin_lock+0x21/0x30 [ 9144.127281] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9144.134788] mdc_close+0x53b/0xd70 [mdc] [ 9144.134846] lmv_close+0x18d/0x490 [lmv] [ 9144.134880] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9144.135008] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9144.135565] ll_md_close+0x274/0x800 [lustre] [ 9144.142975] ? do_raw_spin_unlock+0x75/0x190 [ 9144.142999] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9144.148431] ll_file_release+0x128/0xc50 [lustre] [ 9144.148552] __fput+0x10d/0x440 [ 9144.148563] ____fput+0x12/0x20 [ 9144.148568] task_work_run+0xc6/0x110 [ 9144.148576] exit_to_usermode_loop+0x1dd/0x1f0 [ 9144.148584] do_syscall_64+0x3d6/0x440 [ 9144.148595] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9144.148606] RIP: 0033:0x7f1a63983cd8 [ 9144.148613] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9144.148618] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9144.148624] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9144.148627] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9144.148629] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9144.148631] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9144.148634] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9144.433631] ? do_wait_intr+0xf0/0xf0 [ 9144.436624] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9144.439817] mdc_close+0x53b/0xd70 [mdc] [ 9144.442648] lmv_close+0x18d/0x490 [lmv] [ 9144.445771] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9144.450678] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9144.455162] ll_md_close+0x274/0x800 [lustre] [ 9144.457895] ll_file_release+0x128/0xc50 [lustre] [ 9144.461710] __fput+0x10d/0x440 [ 9144.464432] ____fput+0x12/0x20 [ 9144.466490] task_work_run+0xc6/0x110 [ 9144.468695] exit_to_usermode_loop+0x1dd/0x1f0 [ 9144.471741] do_syscall_64+0x3d6/0x440 [ 9144.474551] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9144.478494] RIP: 0033:0x7f1a63983cd8 [ 9144.480802] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9144.492608] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9144.497404] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9144.501901] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9144.509806] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9144.515820] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9144.519645] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9156.059159] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 9156.062836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9156.086325] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9156.092428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9156.096901] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9156.100067] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9156.110512] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9156.113981] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9156.117582] RDX: 000000000000479c RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9156.121476] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9156.125394] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9156.128665] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9156.132463] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9156.136415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9156.139514] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9156.142738] Call Trace: [ 9156.144212] [ 9156.145222] ? show_regs.cold.9+0x22/0x2f [ 9156.146737] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9156.148649] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9156.150996] ? watchdog+0x40/0x40 [ 9156.152714] ? hrtimer_interrupt+0x128/0x2c0 [ 9156.155233] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9156.157686] ? apic_timer_interrupt+0xf/0x20 [ 9156.160280] [ 9156.161123] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9156.163669] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9156.166411] do_raw_spin_lock+0x177/0x190 [ 9156.170176] _raw_spin_lock+0x21/0x30 [ 9156.171755] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9156.179027] obd_mod_free+0x46/0x140 [ptlrpc] [ 9156.184916] mdc_commit_open+0x91/0xb0 [mdc] [ 9156.188472] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9156.190780] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9156.197210] after_reply+0xe2c/0x1780 [ptlrpc] [ 9156.199124] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9156.201283] ? __next_timer_interrupt+0x160/0x160 [ 9156.204681] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9156.207157] ? do_wait_intr+0xf0/0xf0 [ 9156.208799] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9156.211415] kthread+0x1d1/0x200 [ 9156.212786] ? set_kthread_struct+0x70/0x70 [ 9156.215125] ret_from_fork+0x1f/0x30 [ 9168.056576] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9168.067115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9168.121099] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9168.129307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9168.137165] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9168.143716] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9168.161879] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9168.170286] RAX: 0000000000002684 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9168.172499] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9168.177754] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9168.188078] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9168.195499] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9168.203225] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9168.212713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9168.217259] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9168.223820] Call Trace: [ 9168.228217] [ 9168.229442] ? show_regs.cold.9+0x22/0x2f [ 9168.235107] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9168.238697] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9168.242316] ? watchdog+0x40/0x40 [ 9168.248377] ? hrtimer_interrupt+0x128/0x2c0 [ 9168.250357] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9168.255693] ? apic_timer_interrupt+0xf/0x20 [ 9168.259118] [ 9168.262184] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9168.266477] do_raw_spin_lock+0x177/0x190 [ 9168.271904] _raw_spin_lock+0x21/0x30 [ 9168.273865] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9168.281210] ? do_raw_spin_unlock+0x75/0x190 [ 9168.285753] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9168.290233] ? do_wait_intr+0xf0/0xf0 [ 9168.299578] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9168.307327] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9168.310712] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9168.317584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9168.322916] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9168.329792] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9168.338462] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9168.341881] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9168.347154] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9168.352630] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9168.355606] lmv_intent_open+0x338/0x1150 [lmv] [ 9168.361635] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9168.367796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9168.371683] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9168.377547] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9168.383151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9168.389492] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9168.395701] ? do_raw_spin_unlock+0x75/0x190 [ 9168.399348] ? _raw_spin_unlock+0x12/0x30 [ 9168.405111] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9168.410381] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9168.414830] ll_atomic_open+0x36a/0xd70 [lustre] [ 9168.419323] lookup_open+0x356/0xb40 [ 9168.423225] do_last+0x313/0xfc0 [ 9168.425302] ? nd_jump_root+0xe5/0x160 [ 9168.427317] ? path_init+0x437/0x520 [ 9168.429097] path_openat+0xf7/0x500 [ 9168.430471] ? do_raw_spin_unlock+0x75/0x190 [ 9168.433136] do_filp_open+0x99/0x140 [ 9168.435109] ? getname_flags+0x6e/0x330 [ 9168.437269] ? __check_object_size+0xff/0x256 [ 9168.439713] ? do_raw_spin_unlock+0x75/0x190 [ 9168.442126] ? _raw_spin_unlock+0x12/0x30 [ 9168.444374] do_sys_openat2+0x2b4/0x410 [ 9168.446362] do_sys_open+0x73/0xa0 [ 9168.448562] __x64_sys_openat+0x24/0x30 [ 9168.450423] do_syscall_64+0xc1/0x440 [ 9168.452554] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9168.455254] RIP: 0033:0x7f1a63983332 [ 9168.457243] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9168.484129] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9168.494725] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9168.503476] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9168.510838] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9168.518622] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9168.526046] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9170.921423] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7147s! [ 9170.931799] Showing busy workqueues and worker pools: [ 9170.936420] workqueue events: flags=0x0 [ 9170.941116] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9170.947355] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9170.952663] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9170.958449] pending: sec_gc_main [ptlrpc] [ 9170.964296] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9170.968538] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9170.975244] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9170.978854] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9170.986636] workqueue events_long: flags=0x0 [ 9170.991950] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9170.997739] pending: lu_objects_flush [obdclass] [ 9171.001359] workqueue events_unbound: flags=0x2 [ 9171.005491] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9171.011794] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9171.020180] workqueue events_power_efficient: flags=0x80 [ 9171.026465] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9171.032594] pending: neigh_periodic_work, do_cache_clean [ 9171.035217] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9171.041411] pending: neigh_periodic_work, check_lifetime [ 9171.047126] workqueue mm_percpu_wq: flags=0x8 [ 9171.051730] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9171.058334] pending: vmstat_update [ 9171.060548] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9171.066350] pending: vmstat_update [ 9171.068573] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9171.071463] pending: vmstat_update [ 9171.073090] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9171.076106] pending: vmstat_update [ 9171.077749] workqueue writeback: flags=0x4a [ 9171.079778] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9171.083063] pending: wb_workfn, wb_workfn [ 9171.085312] workqueue xprtiod: flags=0xa [ 9171.087439] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9171.091324] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9171.098473] workqueue ipv6_addrconf: flags=0x40008 [ 9171.101257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9171.113471] pending: addrconf_verify_work [ 9171.116464] workqueue knbd0-recv: flags=0x1a [ 9171.122130] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9171.126896] in-flight: 647:recv_work [ 9171.131890] workqueue ptlrpc_pinger: flags=0x2 [ 9171.135071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9171.138699] pending: ptlrpc_pinger_main [ptlrpc] [ 9171.142557] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9161s workers=2 idle: 652 [ 9172.056232] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 9172.058713] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 9172.065029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9172.065133] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9172.065141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9172.065144] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9172.065159] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9172.065163] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9172.065169] RAX: 000000000000474b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9172.065171] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9172.065174] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9172.065176] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9172.065179] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9172.065183] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9172.065186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9172.065189] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9172.065198] Call Trace: [ 9172.065203] [ 9172.065209] ? show_regs.cold.9+0x22/0x2f [ 9172.065219] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9172.065226] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9172.065229] ? watchdog+0x40/0x40 [ 9172.065234] ? hrtimer_interrupt+0x128/0x2c0 [ 9172.065238] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9172.065246] ? apic_timer_interrupt+0xf/0x20 [ 9172.065251] [ 9172.065254] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9172.065260] do_raw_spin_lock+0x177/0x190 [ 9172.065265] _raw_spin_lock+0x21/0x30 [ 9172.065270] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9172.073471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9172.073570] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9172.073578] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9172.073581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9172.073596] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9172.073601] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9172.073605] RAX: 0000000000007a81 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9172.073608] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9172.073611] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9172.073613] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9172.073617] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9172.073620] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9172.073624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9172.073626] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9172.073636] Call Trace: [ 9172.073641] [ 9172.073647] ? show_regs.cold.9+0x22/0x2f [ 9172.073658] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9172.073665] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9172.073669] ? watchdog+0x40/0x40 [ 9172.073673] ? hrtimer_interrupt+0x128/0x2c0 [ 9172.073678] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9172.073686] ? apic_timer_interrupt+0xf/0x20 [ 9172.073690] [ 9172.073693] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9172.073699] do_raw_spin_lock+0x177/0x190 [ 9172.073704] _raw_spin_lock+0x21/0x30 [ 9172.073709] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9172.074036] ? do_raw_spin_unlock+0x75/0x190 [ 9172.074043] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9172.074374] ? do_wait_intr+0xf0/0xf0 [ 9172.074381] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9172.074574] mdc_close+0x53b/0xd70 [mdc] [ 9172.074611] lmv_close+0x18d/0x490 [lmv] [ 9172.074642] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9172.074754] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9172.074902] ll_md_close+0x274/0x800 [lustre] [ 9172.075145] ll_file_release+0x128/0xc50 [lustre] [ 9172.075236] __fput+0x10d/0x440 [ 9172.075246] ____fput+0x12/0x20 [ 9172.075250] task_work_run+0xc6/0x110 [ 9172.075259] exit_to_usermode_loop+0x1dd/0x1f0 [ 9172.075267] do_syscall_64+0x3d6/0x440 [ 9172.075272] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9172.075279] RIP: 0033:0x7f1a63983cd8 [ 9172.075286] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9172.075290] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9172.075295] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9172.075298] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9172.075300] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9172.075302] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9172.075305] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9172.603160] ? do_raw_spin_unlock+0x75/0x190 [ 9172.604461] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9172.606164] ? do_wait_intr+0xf0/0xf0 [ 9172.607250] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9172.609122] mdc_close+0x53b/0xd70 [mdc] [ 9172.610328] lmv_close+0x18d/0x490 [lmv] [ 9172.611481] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9172.613310] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9172.615196] ll_md_close+0x274/0x800 [lustre] [ 9172.616926] ll_file_release+0x128/0xc50 [lustre] [ 9172.618224] __fput+0x10d/0x440 [ 9172.618839] ____fput+0x12/0x20 [ 9172.619926] task_work_run+0xc6/0x110 [ 9172.621015] exit_to_usermode_loop+0x1dd/0x1f0 [ 9172.622472] do_syscall_64+0x3d6/0x440 [ 9172.624228] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9172.626300] RIP: 0033:0x7f1a63983cd8 [ 9172.627790] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9172.641913] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9172.645938] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9172.651780] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9172.657751] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9172.661558] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9172.665899] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9184.058018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 9184.061702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9184.083426] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9184.088424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9184.092198] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9184.094532] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9184.102716] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9184.106108] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9184.109247] RDX: 0000000000003d7e RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9184.112357] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9184.115239] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9184.118028] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9184.123490] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9184.126915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9184.130093] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9184.133487] Call Trace: [ 9184.134620] [ 9184.135979] ? show_regs.cold.9+0x22/0x2f [ 9184.137703] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9184.140169] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9184.142357] ? watchdog+0x40/0x40 [ 9184.155675] ? hrtimer_interrupt+0x128/0x2c0 [ 9184.161537] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9184.165380] ? apic_timer_interrupt+0xf/0x20 [ 9184.167441] [ 9184.174699] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9184.180432] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9184.183406] do_raw_spin_lock+0x177/0x190 [ 9184.185669] _raw_spin_lock+0x21/0x30 [ 9184.187648] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9184.192239] obd_mod_free+0x46/0x140 [ptlrpc] [ 9184.195022] mdc_commit_open+0x91/0xb0 [mdc] [ 9184.198235] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9184.201802] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9184.207379] after_reply+0xe2c/0x1780 [ptlrpc] [ 9184.212750] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9184.218552] ? __next_timer_interrupt+0x160/0x160 [ 9184.221789] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9184.224530] ? do_wait_intr+0xf0/0xf0 [ 9184.226449] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9184.229527] kthread+0x1d1/0x200 [ 9184.231221] ? set_kthread_struct+0x70/0x70 [ 9184.233125] ret_from_fork+0x1f/0x30 [ 9196.057593] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9196.066371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9196.153414] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9196.164032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9196.174034] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9196.183791] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9196.202067] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9196.212161] RAX: 0000000000006493 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9196.218572] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9196.225802] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9196.230041] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9196.233365] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9196.236955] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9196.240969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9196.243919] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9196.246983] Call Trace: [ 9196.248095] [ 9196.249151] ? show_regs.cold.9+0x22/0x2f [ 9196.254183] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9196.257081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9196.259711] ? watchdog+0x40/0x40 [ 9196.261193] ? hrtimer_interrupt+0x128/0x2c0 [ 9196.263694] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9196.267977] ? apic_timer_interrupt+0xf/0x20 [ 9196.271140] [ 9196.273157] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9196.279075] do_raw_spin_lock+0x177/0x190 [ 9196.284210] _raw_spin_lock+0x21/0x30 [ 9196.285696] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9196.293941] ? do_raw_spin_unlock+0x75/0x190 [ 9196.297087] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9196.299236] ? do_wait_intr+0xf0/0xf0 [ 9196.304998] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9196.310765] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9196.314092] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9196.319357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9196.323584] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9196.331100] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9196.338075] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9196.341158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9196.345329] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9196.351344] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9196.357812] lmv_intent_open+0x338/0x1150 [lmv] [ 9196.359262] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9196.363347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9196.368277] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9196.374687] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9196.379638] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9196.383534] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9196.385615] ? do_raw_spin_unlock+0x75/0x190 [ 9196.387488] ? _raw_spin_unlock+0x12/0x30 [ 9196.389439] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9196.392989] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9196.398373] ll_atomic_open+0x36a/0xd70 [lustre] [ 9196.402094] lookup_open+0x356/0xb40 [ 9196.408669] do_last+0x313/0xfc0 [ 9196.410701] ? nd_jump_root+0xe5/0x160 [ 9196.413452] ? path_init+0x437/0x520 [ 9196.416235] path_openat+0xf7/0x500 [ 9196.417863] ? do_raw_spin_unlock+0x75/0x190 [ 9196.423281] do_filp_open+0x99/0x140 [ 9196.424811] ? getname_flags+0x6e/0x330 [ 9196.431157] ? __check_object_size+0xff/0x256 [ 9196.434553] ? do_raw_spin_unlock+0x75/0x190 [ 9196.440314] ? _raw_spin_unlock+0x12/0x30 [ 9196.441707] do_sys_openat2+0x2b4/0x410 [ 9196.447045] do_sys_open+0x73/0xa0 [ 9196.452426] __x64_sys_openat+0x24/0x30 [ 9196.454379] do_syscall_64+0xc1/0x440 [ 9196.458059] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9196.462461] RIP: 0033:0x7f1a63983332 [ 9196.465435] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9196.481226] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9196.489045] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9196.496983] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9196.503640] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9196.510999] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9196.514214] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9200.058617] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 9200.060295] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 9200.062535] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9200.064002] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9200.064009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9200.064012] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9200.064027] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9200.064032] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9200.064037] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9200.064039] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9200.064041] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9200.064044] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9200.064046] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9200.064050] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9200.064053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9200.064056] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9200.064065] Call Trace: [ 9200.064070] [ 9200.064075] ? show_regs.cold.9+0x22/0x2f [ 9200.064085] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9200.064092] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9200.064097] ? watchdog+0x40/0x40 [ 9200.064101] ? hrtimer_interrupt+0x128/0x2c0 [ 9200.064106] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9200.064113] ? apic_timer_interrupt+0xf/0x20 [ 9200.064118] [ 9200.064121] ? native_safe_halt+0x2e/0x50 [ 9200.064125] kvm_wait+0xcd/0xe0 [ 9200.064133] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9200.064141] do_raw_spin_lock+0x177/0x190 [ 9200.064146] _raw_spin_lock+0x21/0x30 [ 9200.064151] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9200.071995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9200.072063] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9200.072068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9200.072071] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9200.072081] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9200.072084] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9200.072087] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9200.072089] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9200.072091] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9200.072092] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9200.072094] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9200.072097] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9200.072100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9200.072103] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9200.072111] Call Trace: [ 9200.072115] [ 9200.072121] ? show_regs.cold.9+0x22/0x2f [ 9200.072132] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9200.072138] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9200.072142] ? watchdog+0x40/0x40 [ 9200.072147] ? hrtimer_interrupt+0x128/0x2c0 [ 9200.072151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9200.072156] ? apic_timer_interrupt+0xf/0x20 [ 9200.072160] [ 9200.072162] ? native_safe_halt+0x2e/0x50 [ 9200.072164] kvm_wait+0xcd/0xe0 [ 9200.072170] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9200.072177] do_raw_spin_lock+0x177/0x190 [ 9200.072182] _raw_spin_lock+0x21/0x30 [ 9200.072186] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9200.075141] ? do_raw_spin_unlock+0x75/0x190 [ 9200.075150] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9200.100148] ? do_raw_spin_unlock+0x75/0x190 [ 9200.100169] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9200.106039] ? do_wait_intr+0xf0/0xf0 [ 9200.106062] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9200.106494] mdc_close+0x53b/0xd70 [mdc] [ 9200.106539] lmv_close+0x18d/0x490 [lmv] [ 9200.106573] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9200.106816] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9200.106907] ll_md_close+0x274/0x800 [lustre] [ 9200.106996] ll_file_release+0x128/0xc50 [lustre] [ 9200.107087] __fput+0x10d/0x440 [ 9200.107096] ____fput+0x12/0x20 [ 9200.107101] task_work_run+0xc6/0x110 [ 9200.107109] exit_to_usermode_loop+0x1dd/0x1f0 [ 9200.107118] do_syscall_64+0x3d6/0x440 [ 9200.107123] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9200.107133] RIP: 0033:0x7f1a63983cd8 [ 9200.107183] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9200.107189] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9200.107194] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9200.107197] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9200.107199] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9200.107201] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9200.107203] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9200.111616] ? do_wait_intr+0xf0/0xf0 [ 9200.111636] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9200.366956] mdc_close+0x53b/0xd70 [mdc] [ 9200.369072] lmv_close+0x18d/0x490 [lmv] [ 9200.370848] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9200.373412] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9200.375750] ll_md_close+0x274/0x800 [lustre] [ 9200.377879] ll_file_release+0x128/0xc50 [lustre] [ 9200.380421] __fput+0x10d/0x440 [ 9200.381681] ____fput+0x12/0x20 [ 9200.383185] task_work_run+0xc6/0x110 [ 9200.385522] exit_to_usermode_loop+0x1dd/0x1f0 [ 9200.387547] do_syscall_64+0x3d6/0x440 [ 9200.389106] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9200.391533] RIP: 0033:0x7f1a63983cd8 [ 9200.393884] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9200.402750] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9200.406551] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9200.409263] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9200.412859] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9200.416069] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9200.419694] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9201.633118] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7178s! [ 9201.637260] Showing busy workqueues and worker pools: [ 9201.640157] workqueue events: flags=0x0 [ 9201.642218] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9201.645511] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9201.650508] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9201.654095] pending: sec_gc_main [ptlrpc] [ 9201.656790] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9201.660366] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9201.665839] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9201.669465] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9201.674775] workqueue events_long: flags=0x0 [ 9201.676762] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9201.680121] pending: lu_objects_flush [obdclass] [ 9201.683102] workqueue events_unbound: flags=0x2 [ 9201.685031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9201.688269] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9201.692509] workqueue events_power_efficient: flags=0x80 [ 9201.694508] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9201.697533] pending: neigh_periodic_work, do_cache_clean [ 9201.699647] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9201.702256] pending: neigh_periodic_work, check_lifetime [ 9201.704337] workqueue mm_percpu_wq: flags=0x8 [ 9201.706652] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9201.709167] pending: vmstat_update [ 9201.710630] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9201.713273] pending: vmstat_update [ 9201.714985] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9201.717400] pending: vmstat_update [ 9201.718733] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9201.721478] pending: vmstat_update [ 9201.723544] workqueue writeback: flags=0x4a [ 9201.725239] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9201.727741] pending: wb_workfn, wb_workfn [ 9201.729521] workqueue xprtiod: flags=0xa [ 9201.731162] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9201.733886] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9201.737216] workqueue ipv6_addrconf: flags=0x40008 [ 9201.739292] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9201.742255] pending: addrconf_verify_work [ 9201.744289] workqueue knbd0-recv: flags=0x1a [ 9201.746845] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9201.752399] in-flight: 647:recv_work [ 9201.754298] workqueue ptlrpc_pinger: flags=0x2 [ 9201.756225] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9201.759232] pending: ptlrpc_pinger_main [ptlrpc] [ 9201.761438] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9191s workers=2 idle: 652 [ 9212.058089] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9212.063431] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9212.096436] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9212.104039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9212.110551] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9212.115355] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9212.128956] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9212.136033] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9212.141992] RDX: 0000000000000156 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9212.148647] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9212.155091] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9212.160320] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9212.165882] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9212.172417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9212.176965] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9212.182314] Call Trace: [ 9212.184217] [ 9212.185937] ? show_regs.cold.9+0x22/0x2f [ 9212.188617] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9212.191755] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9212.195310] ? watchdog+0x40/0x40 [ 9212.197539] ? hrtimer_interrupt+0x128/0x2c0 [ 9212.201083] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9212.210910] ? apic_timer_interrupt+0xf/0x20 [ 9212.214045] [ 9212.217383] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9212.219996] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9212.222785] do_raw_spin_lock+0x177/0x190 [ 9212.224744] _raw_spin_lock+0x21/0x30 [ 9212.226475] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9212.229185] obd_mod_free+0x46/0x140 [ptlrpc] [ 9212.231800] mdc_commit_open+0x91/0xb0 [mdc] [ 9212.233538] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9212.236752] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9212.240096] after_reply+0xe2c/0x1780 [ptlrpc] [ 9212.242773] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9212.245718] ? __next_timer_interrupt+0x160/0x160 [ 9212.247967] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9212.250226] ? do_wait_intr+0xf0/0xf0 [ 9212.252242] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9212.254806] kthread+0x1d1/0x200 [ 9212.256406] ? set_kthread_struct+0x70/0x70 [ 9212.258086] ret_from_fork+0x1f/0x30 [ 9228.053030] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 9228.055979] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9228.056306] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 9228.057124] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 9228.057165] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9228.057532] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9228.057538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9228.057541] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.057607] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9228.057611] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9228.057615] RAX: 000000000000702e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9228.057617] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9228.057619] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9228.057621] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9228.057623] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9228.057639] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9228.057643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9228.057645] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9228.057652] Call Trace: [ 9228.057657] [ 9228.057666] ? show_regs.cold.9+0x22/0x2f [ 9228.057681] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9228.057691] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9228.057698] ? watchdog+0x40/0x40 [ 9228.057703] ? hrtimer_interrupt+0x128/0x2c0 [ 9228.057707] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9228.057719] ? apic_timer_interrupt+0xf/0x20 [ 9228.057728] [ 9228.057731] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.057740] do_raw_spin_lock+0x177/0x190 [ 9228.057748] _raw_spin_lock+0x21/0x30 [ 9228.057757] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9228.058120] ? do_raw_spin_unlock+0x75/0x190 [ 9228.058129] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9228.058769] ? do_wait_intr+0xf0/0xf0 [ 9228.058784] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9228.059048] mdc_close+0x53b/0xd70 [mdc] [ 9228.059095] lmv_close+0x18d/0x490 [lmv] [ 9228.059136] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9228.059472] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9228.059776] ll_md_close+0x274/0x800 [lustre] [ 9228.059880] ll_file_release+0x128/0xc50 [lustre] [ 9228.059966] __fput+0x10d/0x440 [ 9228.059974] ____fput+0x12/0x20 [ 9228.059983] task_work_run+0xc6/0x110 [ 9228.059992] exit_to_usermode_loop+0x1dd/0x1f0 [ 9228.060003] do_syscall_64+0x3d6/0x440 [ 9228.060011] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9228.060019] RIP: 0033:0x7f1a63983cd8 [ 9228.060026] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9228.060031] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9228.060037] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9228.060039] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9228.060041] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9228.060043] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9228.060049] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9228.129262] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9228.129278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9228.129281] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9228.129295] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9228.129300] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9228.129305] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9228.129307] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9228.129309] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9228.129311] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9228.129313] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9228.129317] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9228.129319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9228.129321] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9228.129331] Call Trace: [ 9228.129336] [ 9228.129342] ? show_regs.cold.9+0x22/0x2f [ 9228.129353] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9228.129360] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9228.129364] ? watchdog+0x40/0x40 [ 9228.129368] ? hrtimer_interrupt+0x128/0x2c0 [ 9228.129371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9228.129377] ? apic_timer_interrupt+0xf/0x20 [ 9228.129382] [ 9228.129385] ? native_safe_halt+0x2e/0x50 [ 9228.129389] kvm_wait+0xcd/0xe0 [ 9228.129396] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9228.129405] do_raw_spin_lock+0x177/0x190 [ 9228.129410] _raw_spin_lock+0x21/0x30 [ 9228.129415] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9228.137939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9228.138020] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9228.138028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9228.138032] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.138047] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9228.138051] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9228.138056] RAX: 000000000000441e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9228.138059] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9228.138061] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9228.138063] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9228.138065] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9228.138069] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9228.138072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9228.138074] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9228.138084] Call Trace: [ 9228.138090] [ 9228.138096] ? show_regs.cold.9+0x22/0x2f [ 9228.138107] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9228.138843] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9228.138848] ? watchdog+0x40/0x40 [ 9228.138853] ? hrtimer_interrupt+0x128/0x2c0 [ 9228.138857] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9228.138866] ? apic_timer_interrupt+0xf/0x20 [ 9228.138871] [ 9228.138874] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.138880] do_raw_spin_lock+0x177/0x190 [ 9228.138885] _raw_spin_lock+0x21/0x30 [ 9228.138891] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9228.147610] ? do_raw_spin_unlock+0x75/0x190 [ 9228.147633] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9228.147909] ? do_wait_intr+0xf0/0xf0 [ 9228.147920] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9228.148701] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9228.148940] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9228.149536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9228.149649] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9228.149686] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9228.149716] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9228.149746] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9228.149841] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9228.150069] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9228.150100] lmv_intent_open+0x338/0x1150 [lmv] [ 9228.150715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9228.150809] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9228.150899] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9228.150924] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9228.151010] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9228.151782] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9228.151868] ? do_raw_spin_unlock+0x75/0x190 [ 9228.151876] ? _raw_spin_unlock+0x12/0x30 [ 9228.151881] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9228.151905] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9228.151924] ll_atomic_open+0x36a/0xd70 [lustre] [ 9228.152008] lookup_open+0x356/0xb40 [ 9228.152017] do_last+0x313/0xfc0 [ 9228.152023] ? nd_jump_root+0xe5/0x160 [ 9228.152027] ? path_init+0x437/0x520 [ 9228.152032] path_openat+0xf7/0x500 [ 9228.152037] ? do_raw_spin_unlock+0x75/0x190 [ 9228.152041] do_filp_open+0x99/0x140 [ 9228.152047] ? getname_flags+0x6e/0x330 [ 9228.152051] ? __check_object_size+0xff/0x256 [ 9228.152057] ? do_raw_spin_unlock+0x75/0x190 [ 9228.152062] ? _raw_spin_unlock+0x12/0x30 [ 9228.152066] do_sys_openat2+0x2b4/0x410 [ 9228.152071] do_sys_open+0x73/0xa0 [ 9228.152075] __x64_sys_openat+0x24/0x30 [ 9228.152079] do_syscall_64+0xc1/0x440 [ 9228.152085] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9228.152092] RIP: 0033:0x7f1a63983332 [ 9228.152100] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9228.152104] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9228.152109] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9228.196840] ? do_raw_spin_unlock+0x75/0x190 [ 9228.196862] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9228.197151] ? do_wait_intr+0xf0/0xf0 [ 9228.197160] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9228.200731] mdc_close+0x53b/0xd70 [mdc] [ 9228.200769] lmv_close+0x18d/0x490 [lmv] [ 9228.200802] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9228.200914] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9228.200996] ll_md_close+0x274/0x800 [lustre] [ 9228.201076] ll_file_release+0x128/0xc50 [lustre] [ 9228.204279] __fput+0x10d/0x440 [ 9228.214320] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9228.214331] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9228.214334] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9228.214336] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9229.240434] ____fput+0x12/0x20 [ 9229.241764] task_work_run+0xc6/0x110 [ 9229.244553] exit_to_usermode_loop+0x1dd/0x1f0 [ 9229.247161] do_syscall_64+0x3d6/0x440 [ 9229.249573] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9229.253873] RIP: 0033:0x7f1a63983cd8 [ 9229.256053] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9229.279471] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9229.294210] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9229.302827] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9229.315906] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9229.325248] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9229.329908] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9231.846552] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7208s! [ 9231.854772] Showing busy workqueues and worker pools: [ 9231.857101] workqueue events: flags=0x0 [ 9231.859969] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9231.866363] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9231.873373] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9231.877210] pending: sec_gc_main [ptlrpc] [ 9231.879727] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9231.882672] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9231.889839] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9231.895770] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9231.903843] workqueue events_long: flags=0x0 [ 9231.908783] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9231.915390] pending: lu_objects_flush [obdclass] [ 9231.917570] workqueue events_unbound: flags=0x2 [ 9231.922456] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9231.926327] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9231.935590] workqueue events_power_efficient: flags=0x80 [ 9231.938696] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9231.941769] pending: neigh_periodic_work, do_cache_clean [ 9231.943388] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9231.946520] pending: neigh_periodic_work, check_lifetime [ 9231.949033] workqueue mm_percpu_wq: flags=0x8 [ 9231.951990] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9231.955122] pending: vmstat_update [ 9231.956894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9231.959844] pending: vmstat_update [ 9231.961696] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9231.964463] pending: vmstat_update [ 9231.966502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9231.969327] pending: vmstat_update [ 9231.970921] workqueue writeback: flags=0x4a [ 9231.972782] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9231.975686] pending: wb_workfn, wb_workfn [ 9231.977456] workqueue xprtiod: flags=0xa [ 9231.978995] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9231.981974] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9231.990680] workqueue ipv6_addrconf: flags=0x40008 [ 9231.993086] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9231.996392] pending: addrconf_verify_work [ 9231.998248] workqueue knbd0-recv: flags=0x1a [ 9232.000207] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9232.003071] in-flight: 647:recv_work [ 9232.004169] workqueue ptlrpc_pinger: flags=0x2 [ 9232.005305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9232.008286] pending: ptlrpc_pinger_main [ptlrpc] [ 9232.015975] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9222s workers=2 idle: 652 [ 9240.058026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9240.060991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9240.092940] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9240.098496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9240.103414] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9240.104958] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9240.113713] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9240.117133] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9240.119616] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9240.121485] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9240.124897] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9240.127215] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9240.129433] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9240.131495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9240.134048] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9240.137567] Call Trace: [ 9240.138186] [ 9240.138663] ? show_regs.cold.9+0x22/0x2f [ 9240.139720] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9240.141559] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9240.142752] ? watchdog+0x40/0x40 [ 9240.146391] ? hrtimer_interrupt+0x128/0x2c0 [ 9240.147583] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9240.149536] ? apic_timer_interrupt+0xf/0x20 [ 9240.150627] [ 9240.151549] ? native_safe_halt+0x2e/0x50 [ 9240.153699] kvm_wait+0xcd/0xe0 [ 9240.154665] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9240.156156] do_raw_spin_lock+0x177/0x190 [ 9240.157719] _raw_spin_lock+0x21/0x30 [ 9240.158691] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9240.160164] obd_mod_free+0x46/0x140 [ptlrpc] [ 9240.162160] mdc_commit_open+0x91/0xb0 [mdc] [ 9240.163761] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9240.165670] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9240.167160] after_reply+0xe2c/0x1780 [ptlrpc] [ 9240.168529] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9240.170294] ? __next_timer_interrupt+0x160/0x160 [ 9240.172276] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9240.174152] ? do_wait_intr+0xf0/0xf0 [ 9240.175599] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9240.177701] kthread+0x1d1/0x200 [ 9240.178970] ? set_kthread_struct+0x70/0x70 [ 9240.180869] ret_from_fork+0x1f/0x30 [ 9256.056488] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9256.057197] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9256.057392] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9256.057400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9256.057468] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9256.057473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9256.057476] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.057490] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9256.057494] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9256.057498] RAX: 0000000000004d11 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9256.057501] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9256.057504] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9256.057506] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9256.057508] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9256.057512] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9256.057514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9256.057517] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9256.057526] Call Trace: [ 9256.057531] [ 9256.057536] ? show_regs.cold.9+0x22/0x2f [ 9256.057547] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9256.057554] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9256.057558] ? watchdog+0x40/0x40 [ 9256.057562] ? hrtimer_interrupt+0x128/0x2c0 [ 9256.057567] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9256.057574] ? apic_timer_interrupt+0xf/0x20 [ 9256.057579] [ 9256.057582] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.057588] do_raw_spin_lock+0x177/0x190 [ 9256.057593] _raw_spin_lock+0x21/0x30 [ 9256.057598] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9256.057870] ? do_raw_spin_unlock+0x75/0x190 [ 9256.057876] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9256.058074] ? do_wait_intr+0xf0/0xf0 [ 9256.058082] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9256.058302] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9256.058578] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9256.059562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9256.059675] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9256.059713] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9256.059742] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9256.059769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9256.059852] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9256.060070] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9256.060100] lmv_intent_open+0x338/0x1150 [lmv] [ 9256.060131] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9256.060231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9256.060318] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9256.060355] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9256.060439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9256.060523] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9256.060608] ? do_raw_spin_unlock+0x75/0x190 [ 9256.060616] ? _raw_spin_unlock+0x12/0x30 [ 9256.060622] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9256.060646] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9256.060666] ll_atomic_open+0x36a/0xd70 [lustre] [ 9256.060750] lookup_open+0x356/0xb40 [ 9256.060760] do_last+0x313/0xfc0 [ 9256.060765] ? nd_jump_root+0xe5/0x160 [ 9256.060770] ? path_init+0x437/0x520 [ 9256.060775] path_openat+0xf7/0x500 [ 9256.060780] ? do_raw_spin_unlock+0x75/0x190 [ 9256.060785] do_filp_open+0x99/0x140 [ 9256.060793] ? getname_flags+0x6e/0x330 [ 9256.060798] ? __check_object_size+0xff/0x256 [ 9256.060806] ? do_raw_spin_unlock+0x75/0x190 [ 9256.060810] ? _raw_spin_unlock+0x12/0x30 [ 9256.060816] do_sys_openat2+0x2b4/0x410 [ 9256.060821] do_sys_open+0x73/0xa0 [ 9256.060825] __x64_sys_openat+0x24/0x30 [ 9256.060830] do_syscall_64+0xc1/0x440 [ 9256.060836] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9256.060842] RIP: 0033:0x7f1a63983332 [ 9256.060849] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9256.060853] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9256.060858] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9256.060860] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9256.060863] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9256.060865] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9256.060867] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9256.063308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9256.063398] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9256.063405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9256.063408] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.063420] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9256.063424] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9256.063428] RAX: 0000000000002d85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9256.063431] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9256.063433] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9256.063434] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9256.063437] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9256.063440] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9256.063442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9256.063444] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9256.063451] Call Trace: [ 9256.063456] [ 9256.063461] ? show_regs.cold.9+0x22/0x2f [ 9256.063469] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9256.063476] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9256.063480] ? watchdog+0x40/0x40 [ 9256.063484] ? hrtimer_interrupt+0x128/0x2c0 [ 9256.063488] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9256.063496] ? apic_timer_interrupt+0xf/0x20 [ 9256.063500] [ 9256.063503] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.063508] do_raw_spin_lock+0x177/0x190 [ 9256.063513] _raw_spin_lock+0x21/0x30 [ 9256.063517] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9256.063791] ? do_raw_spin_unlock+0x75/0x190 [ 9256.063796] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9256.064179] ? do_wait_intr+0xf0/0xf0 [ 9256.064187] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9256.064388] mdc_close+0x53b/0xd70 [mdc] [ 9256.064430] lmv_close+0x18d/0x490 [lmv] [ 9256.064463] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9256.064570] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9256.064649] ll_md_close+0x274/0x800 [lustre] [ 9256.064730] ll_file_release+0x128/0xc50 [lustre] [ 9256.064810] __fput+0x10d/0x440 [ 9256.064819] ____fput+0x12/0x20 [ 9256.064823] task_work_run+0xc6/0x110 [ 9256.064831] exit_to_usermode_loop+0x1dd/0x1f0 [ 9256.064838] do_syscall_64+0x3d6/0x440 [ 9256.064842] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9256.064849] RIP: 0033:0x7f1a63983cd8 [ 9256.064856] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9256.064860] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9256.064864] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9256.064867] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9256.064869] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9256.064871] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9256.064873] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9256.843667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9256.872125] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9256.884130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9256.892526] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.896191] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9256.907072] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9256.915287] RAX: 000000000000210a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9256.923852] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9256.933407] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9256.941773] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9256.950110] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9256.959323] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9256.966297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9256.971759] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9256.987270] Call Trace: [ 9256.989052] [ 9256.991942] ? show_regs.cold.9+0x22/0x2f [ 9256.996317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9256.998516] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9257.000794] ? watchdog+0x40/0x40 [ 9257.003087] ? hrtimer_interrupt+0x128/0x2c0 [ 9257.005824] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9257.009462] ? apic_timer_interrupt+0xf/0x20 [ 9257.012629] [ 9257.015694] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9257.024308] do_raw_spin_lock+0x177/0x190 [ 9257.029301] _raw_spin_lock+0x21/0x30 [ 9257.036798] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9257.048738] ? do_raw_spin_unlock+0x75/0x190 [ 9257.056114] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9257.065570] ? do_wait_intr+0xf0/0xf0 [ 9257.070395] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9257.076944] mdc_close+0x53b/0xd70 [mdc] [ 9257.080868] lmv_close+0x18d/0x490 [lmv] [ 9257.082791] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9257.086737] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9257.089993] ll_md_close+0x274/0x800 [lustre] [ 9257.094447] ll_file_release+0x128/0xc50 [lustre] [ 9257.100400] __fput+0x10d/0x440 [ 9257.106829] ____fput+0x12/0x20 [ 9257.111849] task_work_run+0xc6/0x110 [ 9257.114028] exit_to_usermode_loop+0x1dd/0x1f0 [ 9257.126051] do_syscall_64+0x3d6/0x440 [ 9257.129723] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9257.137040] RIP: 0033:0x7f1a63983cd8 [ 9257.141503] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9257.159220] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9257.168095] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9257.170637] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9257.185672] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9257.190429] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9257.194459] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9262.048511] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7239s! [ 9262.058958] Showing busy workqueues and worker pools: [ 9262.062340] workqueue events: flags=0x0 [ 9262.067410] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9262.073759] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9262.086577] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9262.097311] pending: sec_gc_main [ptlrpc] [ 9262.102176] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9262.107962] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9262.115564] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9262.120120] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9262.126713] workqueue events_long: flags=0x0 [ 9262.142176] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9262.151964] pending: lu_objects_flush [obdclass] [ 9262.160170] workqueue events_unbound: flags=0x2 [ 9262.167179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9262.178532] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9262.199156] workqueue events_power_efficient: flags=0x80 [ 9262.207763] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9262.215416] pending: neigh_periodic_work, do_cache_clean [ 9262.225378] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9262.231866] pending: neigh_periodic_work, check_lifetime [ 9262.244503] workqueue mm_percpu_wq: flags=0x8 [ 9262.255383] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9262.270678] pending: vmstat_update [ 9262.278794] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9262.289843] pending: vmstat_update [ 9262.291568] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9262.298712] pending: vmstat_update [ 9262.300166] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9262.309744] pending: vmstat_update [ 9262.313539] workqueue writeback: flags=0x4a [ 9262.317857] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9262.323678] pending: wb_workfn, wb_workfn [ 9262.332425] workqueue xprtiod: flags=0xa [ 9262.333745] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9262.343479] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9262.355412] workqueue ipv6_addrconf: flags=0x40008 [ 9262.362411] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9262.369598] pending: addrconf_verify_work [ 9262.377136] workqueue knbd0-recv: flags=0x1a [ 9262.381157] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9262.388947] in-flight: 647:recv_work [ 9262.392566] workqueue ptlrpc_pinger: flags=0x2 [ 9262.397906] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9262.404429] pending: ptlrpc_pinger_main [ptlrpc] [ 9262.409857] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9252s workers=2 idle: 652 [ 9268.058098] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9268.066969] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9268.104209] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9268.108010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9268.114509] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9268.121878] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9268.135988] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9268.143292] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9268.145883] RDX: 0000000000005e39 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9268.157301] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9268.161291] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9268.166232] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9268.173911] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9268.178734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9268.190347] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9268.195507] Call Trace: [ 9268.200717] [ 9268.201877] ? show_regs.cold.9+0x22/0x2f [ 9268.203912] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9268.208001] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9268.212313] ? watchdog+0x40/0x40 [ 9268.215170] ? hrtimer_interrupt+0x128/0x2c0 [ 9268.216824] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9268.222892] ? apic_timer_interrupt+0xf/0x20 [ 9268.227437] [ 9268.229274] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9268.235683] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9268.240003] do_raw_spin_lock+0x177/0x190 [ 9268.251807] _raw_spin_lock+0x21/0x30 [ 9268.254916] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9268.259946] obd_mod_free+0x46/0x140 [ptlrpc] [ 9268.261969] mdc_commit_open+0x91/0xb0 [mdc] [ 9268.263873] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9268.265977] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9268.268001] after_reply+0xe2c/0x1780 [ptlrpc] [ 9268.270301] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9268.272200] ? __next_timer_interrupt+0x160/0x160 [ 9268.273817] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9268.275507] ? do_wait_intr+0xf0/0xf0 [ 9268.276585] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9268.282497] kthread+0x1d1/0x200 [ 9268.283304] ? set_kthread_struct+0x70/0x70 [ 9268.284283] ret_from_fork+0x1f/0x30 [ 9282.598013] rcu: INFO: rcu_sched self-detected stall on CPU [ 9282.600601] rcu: 3-...!: (4582294 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 9282.604569] (t=7260197 jiffies g=269633 q=2587) [ 9282.606438] rcu: rcu_sched kthread starved for 7259843 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 9282.610777] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9282.616613] rcu: RCU grace-period kthread stack dump: [ 9282.621945] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9282.629867] Call Trace: [ 9282.631087] __schedule+0x351/0xcb0 [ 9282.634157] schedule+0xc0/0x180 [ 9282.636054] schedule_timeout+0xb4/0x190 [ 9282.638014] ? do_raw_spin_unlock+0x75/0x190 [ 9282.640619] ? __next_timer_interrupt+0x160/0x160 [ 9282.643338] ? prepare_to_swait_event+0xf3/0x140 [ 9282.645551] rcu_gp_kthread+0x8dd/0x15b0 [ 9282.647808] ? force_qs_rnp+0x2c0/0x2c0 [ 9282.649981] kthread+0x1d1/0x200 [ 9282.651330] ? set_kthread_struct+0x70/0x70 [ 9282.652999] ret_from_fork+0x1f/0x30 [ 9282.655624] rcu: Stack dump where RCU GP kthread last ran: [ 9282.659501] Sending NMI from CPU 3 to CPUs 0: [ 9282.671127] NMI backtrace for cpu 0 [ 9282.671135] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9282.671137] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9282.671139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9282.671141] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9282.671143] RSP: 0018:ffffc38541c37500 EFLAGS: 00000216 [ 9282.671145] RAX: 000000000000768e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9282.671147] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9282.671148] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9282.671150] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9282.671152] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9282.671153] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9282.671155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9282.671156] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9282.671158] Call Trace: [ 9282.671158] [ 9282.671160] ? show_regs.cold.9+0x22/0x2f [ 9282.671161] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9282.671162] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9282.671164] ? nmi_handle+0xbd/0x210 [ 9282.671165] ? default_do_nmi+0x4b/0x1b0 [ 9282.671166] ? do_nmi+0x280/0x380 [ 9282.671167] ? end_repeat_nmi+0x16/0x1f [ 9282.671168] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9282.671169] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9282.671170] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9282.671171] [ 9282.671172] do_raw_spin_lock+0x177/0x190 [ 9282.671173] _raw_spin_lock+0x21/0x30 [ 9282.671175] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9282.671176] ? do_raw_spin_unlock+0x75/0x190 [ 9282.671177] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9282.671178] ? do_wait_intr+0xf0/0xf0 [ 9282.671179] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9282.671180] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9282.671182] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9282.671183] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9282.671185] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9282.671186] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9282.671187] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9282.671188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9282.671189] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9282.671191] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9282.671192] lmv_intent_open+0x338/0x1150 [lmv] [ 9282.671193] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9282.671194] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9282.671196] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9282.671197] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9282.671197] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9282.671198] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9282.671199] ? do_raw_spin_unlock+0x75/0x190 [ 9282.671200] ? _raw_spin_unlock+0x12/0x30 [ 9282.671202] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9282.671202] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9282.671204] ll_atomic_open+0x36a/0xd70 [lustre] [ 9282.671204] lookup_open+0x356/0xb40 [ 9282.671205] do_last+0x313/0xfc0 [ 9282.671206] ? nd_jump_root+0xe5/0x160 [ 9282.671207] ? path_init+0x437/0x520 [ 9282.671208] path_openat+0xf7/0x500 [ 9282.671209] ? do_raw_spin_unlock+0x75/0x190 [ 9282.671210] do_filp_open+0x99/0x140 [ 9282.671211] ? getname_flags+0x6e/0x330 [ 9282.671212] ? __check_object_size+0xff/0x256 [ 9282.671213] ? do_raw_spin_unlock+0x75/0x190 [ 9282.671214] ? _raw_spin_unlock+0x12/0x30 [ 9282.671215] do_sys_openat2+0x2b4/0x410 [ 9282.671216] do_sys_open+0x73/0xa0 [ 9282.671217] __x64_sys_openat+0x24/0x30 [ 9282.671218] do_syscall_64+0xc1/0x440 [ 9282.671219] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9282.671220] RIP: 0033:0x7f1a63983332 [ 9282.671222] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9282.671224] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9282.671227] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9282.671228] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9282.671230] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9282.671231] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9282.671233] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9282.673090] NMI backtrace for cpu 3 [ 9283.047594] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9283.052409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9283.055739] Call Trace: [ 9283.057172] [ 9283.057984] ? dump_stack+0xbb/0x10e [ 9283.061673] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9283.064532] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9283.066860] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9283.071662] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9283.076766] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9283.079737] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9283.085456] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9283.089442] ? update_process_times+0x6d/0xc0 [ 9283.092409] ? tick_sched_handle.isra.18+0x27/0x90 [ 9283.102779] ? tick_sched_timer+0x53/0xb0 [ 9283.107654] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9283.111212] ? tick_sched_handle.isra.18+0x90/0x90 [ 9283.116460] ? hrtimer_interrupt+0x128/0x2c0 [ 9283.119335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9283.122081] ? apic_timer_interrupt+0xf/0x20 [ 9283.124176] [ 9283.125405] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9283.127441] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9283.133289] ? do_raw_spin_lock+0x177/0x190 [ 9283.135307] ? _raw_spin_lock+0x21/0x30 [ 9283.136981] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9283.139266] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9283.141634] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9283.143107] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9283.144950] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9283.147441] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9283.149371] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9283.151362] ? __next_timer_interrupt+0x160/0x160 [ 9283.152795] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9283.154479] ? do_wait_intr+0xf0/0xf0 [ 9283.155615] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9283.157663] ? kthread+0x1d1/0x200 [ 9283.158644] ? set_kthread_struct+0x70/0x70 [ 9283.159944] ? ret_from_fork+0x1f/0x30 [ 9284.056025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9284.057023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9284.058775] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9284.058852] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9284.058859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9284.058863] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9284.058938] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9284.058951] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9284.058956] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9284.058959] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9284.058962] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9284.058964] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9284.058966] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9284.058970] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9284.058974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9284.058976] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9284.058985] Call Trace: [ 9284.058990] [ 9284.058995] ? show_regs.cold.9+0x22/0x2f [ 9284.063271] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9284.063341] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9284.063446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9284.063449] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9284.063462] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9284.063465] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9284.063470] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9284.063472] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9284.063475] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9284.063477] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9284.063479] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9284.063483] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9284.063486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9284.063489] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9284.063496] Call Trace: [ 9284.063503] [ 9284.063507] ? show_regs.cold.9+0x22/0x2f [ 9284.063516] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9284.063521] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9284.063525] ? watchdog+0x40/0x40 [ 9284.063529] ? hrtimer_interrupt+0x128/0x2c0 [ 9284.063532] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9284.063538] ? apic_timer_interrupt+0xf/0x20 [ 9284.063542] [ 9284.063544] ? native_safe_halt+0x2e/0x50 [ 9284.063547] kvm_wait+0xcd/0xe0 [ 9284.063552] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9284.063558] do_raw_spin_lock+0x177/0x190 [ 9284.063561] _raw_spin_lock+0x21/0x30 [ 9284.063565] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9284.064642] ? do_raw_spin_unlock+0x75/0x190 [ 9284.064650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9284.065206] ? do_wait_intr+0xf0/0xf0 [ 9284.065215] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9284.065484] mdc_close+0x53b/0xd70 [mdc] [ 9284.065522] lmv_close+0x18d/0x490 [lmv] [ 9284.065555] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9284.065728] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9284.065807] ll_md_close+0x274/0x800 [lustre] [ 9284.065917] ll_file_release+0x128/0xc50 [lustre] [ 9284.066156] __fput+0x10d/0x440 [ 9284.066166] ____fput+0x12/0x20 [ 9284.066171] task_work_run+0xc6/0x110 [ 9284.066179] exit_to_usermode_loop+0x1dd/0x1f0 [ 9284.066186] do_syscall_64+0x3d6/0x440 [ 9284.066191] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9284.066199] RIP: 0033:0x7f1a63983cd8 [ 9284.066206] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9284.066214] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9284.066219] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9284.066221] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9284.066224] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9284.066226] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9284.066228] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9284.454193] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9284.456940] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9284.460940] ? watchdog+0x40/0x40 [ 9284.463078] ? hrtimer_interrupt+0x128/0x2c0 [ 9284.465845] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9284.467947] ? apic_timer_interrupt+0xf/0x20 [ 9284.470112] [ 9284.472355] ? native_safe_halt+0x2e/0x50 [ 9284.475149] kvm_wait+0xcd/0xe0 [ 9284.476836] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9284.481911] do_raw_spin_lock+0x177/0x190 [ 9284.486671] _raw_spin_lock+0x21/0x30 [ 9284.487974] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9284.495262] ? do_raw_spin_unlock+0x75/0x190 [ 9284.497147] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9284.500685] ? do_wait_intr+0xf0/0xf0 [ 9284.505378] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9284.507765] mdc_close+0x53b/0xd70 [mdc] [ 9284.513393] lmv_close+0x18d/0x490 [lmv] [ 9284.516476] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9284.519149] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9284.521412] ll_md_close+0x274/0x800 [lustre] [ 9284.523602] ll_file_release+0x128/0xc50 [lustre] [ 9284.525556] __fput+0x10d/0x440 [ 9284.527569] ____fput+0x12/0x20 [ 9284.528763] task_work_run+0xc6/0x110 [ 9284.530725] exit_to_usermode_loop+0x1dd/0x1f0 [ 9284.532678] do_syscall_64+0x3d6/0x440 [ 9284.535278] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9284.538127] RIP: 0033:0x7f1a63983cd8 [ 9284.539916] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9284.553008] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9284.557369] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9284.562305] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9284.565355] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9284.569746] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9284.575327] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9292.769045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7269s! [ 9292.772341] Showing busy workqueues and worker pools: [ 9292.775020] workqueue events: flags=0x0 [ 9292.777656] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9292.781596] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9292.785690] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9292.788689] pending: sec_gc_main [ptlrpc] [ 9292.791296] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9292.797695] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9292.804131] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9292.807299] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9292.815670] workqueue events_long: flags=0x0 [ 9292.818073] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9292.820966] pending: lu_objects_flush [obdclass] [ 9292.823983] workqueue events_unbound: flags=0x2 [ 9292.826407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9292.828962] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9292.834410] workqueue events_power_efficient: flags=0x80 [ 9292.837209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9292.840877] pending: neigh_periodic_work, do_cache_clean [ 9292.844750] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9292.848586] pending: neigh_periodic_work, check_lifetime [ 9292.852126] workqueue mm_percpu_wq: flags=0x8 [ 9292.855610] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9292.861690] pending: vmstat_update [ 9292.864135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9292.868374] pending: vmstat_update [ 9292.870142] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9292.873596] pending: vmstat_update [ 9292.875361] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9292.878411] pending: vmstat_update [ 9292.880238] workqueue writeback: flags=0x4a [ 9292.882422] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9292.884634] pending: wb_workfn, wb_workfn [ 9292.886089] workqueue xprtiod: flags=0xa [ 9292.887110] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9292.888560] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9292.890412] workqueue ipv6_addrconf: flags=0x40008 [ 9292.891453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9292.892924] pending: addrconf_verify_work [ 9292.893932] workqueue knbd0-recv: flags=0x1a [ 9292.894961] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9292.896466] in-flight: 647:recv_work [ 9292.897613] workqueue ptlrpc_pinger: flags=0x2 [ 9292.898617] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9292.900046] pending: ptlrpc_pinger_main [ptlrpc] [ 9292.901349] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9282s workers=2 idle: 652 [ 9308.058040] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9308.061795] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9308.088025] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9308.093607] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9308.097406] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9308.100192] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9308.108918] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9308.112448] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9308.115816] RDX: 0000000000006b5b RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9308.118578] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9308.121275] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9308.123841] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9308.126283] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9308.129853] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9308.132530] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9308.136927] Call Trace: [ 9308.138134] [ 9308.138940] ? show_regs.cold.9+0x22/0x2f [ 9308.140796] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9308.143123] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9308.145867] ? watchdog+0x40/0x40 [ 9308.147569] ? hrtimer_interrupt+0x128/0x2c0 [ 9308.149872] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9308.152007] ? apic_timer_interrupt+0xf/0x20 [ 9308.155934] [ 9308.157033] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9308.161598] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9308.167000] do_raw_spin_lock+0x177/0x190 [ 9308.169556] _raw_spin_lock+0x21/0x30 [ 9308.172263] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9308.175227] obd_mod_free+0x46/0x140 [ptlrpc] [ 9308.177963] mdc_commit_open+0x91/0xb0 [mdc] [ 9308.180309] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9308.182781] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9308.185577] after_reply+0xe2c/0x1780 [ptlrpc] [ 9308.187718] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9308.189928] ? __next_timer_interrupt+0x160/0x160 [ 9308.192144] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9308.195873] ? do_wait_intr+0xf0/0xf0 [ 9308.198158] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9308.202580] kthread+0x1d1/0x200 [ 9308.206598] ? set_kthread_struct+0x70/0x70 [ 9308.211310] ret_from_fork+0x1f/0x30 [ 9312.056738] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9312.057023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9312.060280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9312.060356] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9312.060364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9312.060367] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.060382] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9312.060387] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9312.060392] RAX: 000000000000532d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9312.060394] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9312.060396] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9312.060398] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9312.060401] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9312.060405] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9312.060407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9312.060409] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9312.060418] Call Trace: [ 9312.060424] [ 9312.060430] ? show_regs.cold.9+0x22/0x2f [ 9312.060668] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9312.060675] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9312.060679] ? watchdog+0x40/0x40 [ 9312.060683] ? hrtimer_interrupt+0x128/0x2c0 [ 9312.060688] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9312.060697] ? apic_timer_interrupt+0xf/0x20 [ 9312.060707] [ 9312.060710] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.060717] do_raw_spin_lock+0x177/0x190 [ 9312.060722] _raw_spin_lock+0x21/0x30 [ 9312.060727] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9312.066359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9312.066452] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9312.066460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9312.066464] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.066479] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9312.066483] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9312.066488] RAX: 00000000000060ac RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9312.066491] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9312.066493] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9312.066495] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9312.066497] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9312.066501] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9312.066504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9312.066506] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9312.066806] Call Trace: [ 9312.090327] ? do_raw_spin_unlock+0x75/0x190 [ 9312.097876] [ 9312.102017] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9312.106829] ? show_regs.cold.9+0x22/0x2f [ 9312.106849] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9312.117056] ? do_wait_intr+0xf0/0xf0 [ 9312.117076] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9312.117388] mdc_close+0x53b/0xd70 [mdc] [ 9312.117527] lmv_close+0x18d/0x490 [lmv] [ 9312.117559] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9312.117858] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9312.117940] ll_md_close+0x274/0x800 [lustre] [ 9312.118034] ll_file_release+0x128/0xc50 [lustre] [ 9312.118115] __fput+0x10d/0x440 [ 9312.118124] ____fput+0x12/0x20 [ 9312.118128] task_work_run+0xc6/0x110 [ 9312.118135] exit_to_usermode_loop+0x1dd/0x1f0 [ 9312.118144] do_syscall_64+0x3d6/0x440 [ 9312.118149] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9312.118159] RIP: 0033:0x7f1a63983cd8 [ 9312.118167] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9312.118172] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9312.118177] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9312.118180] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9312.118182] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9312.118183] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9312.118185] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9312.376377] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9312.379807] ? watchdog+0x40/0x40 [ 9312.382290] ? hrtimer_interrupt+0x128/0x2c0 [ 9312.384312] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9312.387957] ? apic_timer_interrupt+0xf/0x20 [ 9312.390777] [ 9312.391912] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.396500] do_raw_spin_lock+0x177/0x190 [ 9312.399083] _raw_spin_lock+0x21/0x30 [ 9312.401747] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9312.404900] ? do_raw_spin_unlock+0x75/0x190 [ 9312.406338] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9312.409024] ? do_wait_intr+0xf0/0xf0 [ 9312.412472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9312.416114] mdc_close+0x53b/0xd70 [mdc] [ 9312.417810] lmv_close+0x18d/0x490 [lmv] [ 9312.421099] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9312.424342] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9312.428800] ll_md_close+0x274/0x800 [lustre] [ 9312.430862] ll_file_release+0x128/0xc50 [lustre] [ 9312.433295] __fput+0x10d/0x440 [ 9312.434705] ____fput+0x12/0x20 [ 9312.436320] task_work_run+0xc6/0x110 [ 9312.438064] exit_to_usermode_loop+0x1dd/0x1f0 [ 9312.442107] do_syscall_64+0x3d6/0x440 [ 9312.444143] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9312.448319] RIP: 0033:0x7f1a63983cd8 [ 9312.450042] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9312.466281] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9312.474669] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9312.477806] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9312.481526] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9312.484110] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9312.491061] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9320.056413] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9320.060429] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9320.094718] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9320.101388] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9320.110875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9320.115416] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9320.131352] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9320.138415] RAX: 0000000000007b69 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9320.143129] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9320.146206] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9320.149777] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9320.152857] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9320.155977] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9320.159803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9320.162834] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9320.166066] Call Trace: [ 9320.167163] [ 9320.168442] ? show_regs.cold.9+0x22/0x2f [ 9320.170597] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9320.173202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9320.177107] ? watchdog+0x40/0x40 [ 9320.178399] ? hrtimer_interrupt+0x128/0x2c0 [ 9320.186451] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9320.188152] ? apic_timer_interrupt+0xf/0x20 [ 9320.196295] [ 9320.201817] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9320.210318] do_raw_spin_lock+0x177/0x190 [ 9320.212440] _raw_spin_lock+0x21/0x30 [ 9320.215455] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9320.221482] ? do_raw_spin_unlock+0x75/0x190 [ 9320.224051] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9320.226578] ? do_wait_intr+0xf0/0xf0 [ 9320.227912] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9320.243112] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9320.249920] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9320.257005] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9320.264101] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9320.271688] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9320.275121] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9320.280685] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9320.285501] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9320.290497] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9320.294072] lmv_intent_open+0x338/0x1150 [lmv] [ 9320.296311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9320.301442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9320.306240] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9320.309212] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9320.313879] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9320.318998] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9320.322167] ? do_raw_spin_unlock+0x75/0x190 [ 9320.327242] ? _raw_spin_unlock+0x12/0x30 [ 9320.335027] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9320.337612] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9320.340987] ll_atomic_open+0x36a/0xd70 [lustre] [ 9320.343438] lookup_open+0x356/0xb40 [ 9320.347289] do_last+0x313/0xfc0 [ 9320.349234] ? nd_jump_root+0xe5/0x160 [ 9320.352218] ? path_init+0x437/0x520 [ 9320.358945] path_openat+0xf7/0x500 [ 9320.360271] ? do_raw_spin_unlock+0x75/0x190 [ 9320.364407] do_filp_open+0x99/0x140 [ 9320.365978] ? getname_flags+0x6e/0x330 [ 9320.370123] ? __check_object_size+0xff/0x256 [ 9320.373018] ? do_raw_spin_unlock+0x75/0x190 [ 9320.375460] ? _raw_spin_unlock+0x12/0x30 [ 9320.380391] do_sys_openat2+0x2b4/0x410 [ 9320.382373] do_sys_open+0x73/0xa0 [ 9320.386049] __x64_sys_openat+0x24/0x30 [ 9320.390472] do_syscall_64+0xc1/0x440 [ 9320.394490] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9320.396350] RIP: 0033:0x7f1a63983332 [ 9320.403841] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9320.419938] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9320.428414] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9320.436073] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9320.443289] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9320.446299] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9320.449469] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9322.977062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7299s! [ 9322.985505] Showing busy workqueues and worker pools: [ 9322.988859] workqueue events: flags=0x0 [ 9322.991599] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9322.995091] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9323.007130] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9323.016677] pending: sec_gc_main [ptlrpc] [ 9323.020982] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9323.025667] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9323.032180] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9323.037212] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9323.044068] workqueue events_long: flags=0x0 [ 9323.046735] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9323.052710] pending: lu_objects_flush [obdclass] [ 9323.055975] workqueue events_unbound: flags=0x2 [ 9323.059077] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9323.063687] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9323.070182] workqueue events_power_efficient: flags=0x80 [ 9323.074567] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9323.079831] pending: neigh_periodic_work, do_cache_clean [ 9323.083329] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9323.089912] pending: neigh_periodic_work, check_lifetime [ 9323.094224] workqueue mm_percpu_wq: flags=0x8 [ 9323.097917] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9323.102166] pending: vmstat_update [ 9323.104037] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9323.108159] pending: vmstat_update [ 9323.111045] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9323.115969] pending: vmstat_update [ 9323.118163] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9323.122559] pending: vmstat_update [ 9323.125153] workqueue writeback: flags=0x4a [ 9323.127274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9323.131064] pending: wb_workfn, wb_workfn [ 9323.133313] workqueue xprtiod: flags=0xa [ 9323.135119] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9323.139349] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9323.145201] workqueue ipv6_addrconf: flags=0x40008 [ 9323.147957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9323.151712] pending: addrconf_verify_work [ 9323.154861] workqueue knbd0-recv: flags=0x1a [ 9323.157328] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9323.161299] in-flight: 647:recv_work [ 9323.164602] workqueue ptlrpc_pinger: flags=0x2 [ 9323.167621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9323.173833] pending: ptlrpc_pinger_main [ptlrpc] [ 9323.177326] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9313s workers=2 idle: 652 [ 9336.058052] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9336.061783] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9336.091407] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9336.098330] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9336.103164] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9336.106539] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9336.116845] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9336.120317] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9336.123757] RDX: 0000000000006784 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9336.128644] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9336.133505] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9336.137196] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9336.141040] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9336.145024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9336.148163] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9336.152330] Call Trace: [ 9336.153597] [ 9336.154763] ? show_regs.cold.9+0x22/0x2f [ 9336.157234] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9336.160328] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9336.162741] ? watchdog+0x40/0x40 [ 9336.164568] ? hrtimer_interrupt+0x128/0x2c0 [ 9336.166614] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9336.169028] ? apic_timer_interrupt+0xf/0x20 [ 9336.172266] [ 9336.173416] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9336.176001] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9336.178413] do_raw_spin_lock+0x177/0x190 [ 9336.180636] _raw_spin_lock+0x21/0x30 [ 9336.182144] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9336.185150] obd_mod_free+0x46/0x140 [ptlrpc] [ 9336.188151] mdc_commit_open+0x91/0xb0 [mdc] [ 9336.190420] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9336.193204] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9336.196012] after_reply+0xe2c/0x1780 [ptlrpc] [ 9336.198358] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9336.201104] ? __next_timer_interrupt+0x160/0x160 [ 9336.202926] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9336.204848] ? do_wait_intr+0xf0/0xf0 [ 9336.206491] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9336.209041] kthread+0x1d1/0x200 [ 9336.210311] ? set_kthread_struct+0x70/0x70 [ 9336.211883] ret_from_fork+0x1f/0x30 [ 9340.056608] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9340.060271] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9340.064723] Modules linked in: [ 9340.069825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9340.069882] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9340.069888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9340.069890] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.069901] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9340.069905] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9340.069908] RAX: 0000000000001f71 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9340.069909] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9340.069957] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9340.069961] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9340.069963] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9340.069966] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9340.069969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9340.069971] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9340.069979] Call Trace: [ 9340.069985] [ 9340.071769] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9340.071882] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9340.071889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9340.071891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.071903] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9340.071907] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9340.071911] RAX: 0000000000005e86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9340.071913] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9340.071916] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9340.071920] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9340.118521] ? show_regs.cold.9+0x22/0x2f [ 9340.118543] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9340.118550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9340.118555] ? watchdog+0x40/0x40 [ 9340.118559] ? hrtimer_interrupt+0x128/0x2c0 [ 9340.118564] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9340.124001] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9340.124024] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9340.124028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9340.124030] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9340.124039] Call Trace: [ 9340.124052] [ 9340.124057] ? show_regs.cold.9+0x22/0x2f [ 9340.124075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9340.124081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9340.124086] ? watchdog+0x40/0x40 [ 9340.124095] ? hrtimer_interrupt+0x128/0x2c0 [ 9340.124099] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9340.124106] ? apic_timer_interrupt+0xf/0x20 [ 9340.124111] [ 9340.124113] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.124124] do_raw_spin_lock+0x177/0x190 [ 9340.131253] ? apic_timer_interrupt+0xf/0x20 [ 9340.131274] [ 9340.131279] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.134257] _raw_spin_lock+0x21/0x30 [ 9340.134277] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9340.151177] do_raw_spin_lock+0x177/0x190 [ 9340.157252] ? do_raw_spin_unlock+0x75/0x190 [ 9340.164842] _raw_spin_lock+0x21/0x30 [ 9340.175468] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9340.175762] ? do_wait_intr+0xf0/0xf0 [ 9340.175773] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9340.175969] mdc_close+0x53b/0xd70 [mdc] [ 9340.176037] lmv_close+0x18d/0x490 [lmv] [ 9340.176067] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9340.176186] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9340.176267] ll_md_close+0x274/0x800 [lustre] [ 9340.176364] ll_file_release+0x128/0xc50 [lustre] [ 9340.176451] __fput+0x10d/0x440 [ 9340.176460] ____fput+0x12/0x20 [ 9340.176465] task_work_run+0xc6/0x110 [ 9340.176473] exit_to_usermode_loop+0x1dd/0x1f0 [ 9340.176482] do_syscall_64+0x3d6/0x440 [ 9340.176486] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9340.176501] RIP: 0033:0x7f1a63983cd8 [ 9340.176509] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9340.176513] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9340.176518] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9340.176521] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9340.176523] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9340.176525] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9340.176527] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9340.431337] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9340.434518] ? do_raw_spin_unlock+0x75/0x190 [ 9340.436711] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9340.439407] ? do_wait_intr+0xf0/0xf0 [ 9340.441632] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9340.444542] mdc_close+0x53b/0xd70 [mdc] [ 9340.446803] lmv_close+0x18d/0x490 [lmv] [ 9340.448766] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9340.454264] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9340.456903] ll_md_close+0x274/0x800 [lustre] [ 9340.461661] ll_file_release+0x128/0xc50 [lustre] [ 9340.467534] __fput+0x10d/0x440 [ 9340.470491] ____fput+0x12/0x20 [ 9340.471980] task_work_run+0xc6/0x110 [ 9340.474617] exit_to_usermode_loop+0x1dd/0x1f0 [ 9340.479779] do_syscall_64+0x3d6/0x440 [ 9340.481717] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9340.484582] RIP: 0033:0x7f1a63983cd8 [ 9340.488865] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9340.502676] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9340.507789] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9340.513673] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9340.516009] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9340.518771] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9340.520879] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9348.057333] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 9348.063674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9348.111119] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9348.124668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9348.133866] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9348.135955] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9348.161441] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9348.166314] RAX: 0000000000004e58 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9348.174882] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9348.183856] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9348.191054] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9348.199091] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9348.208388] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9348.215489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9348.220715] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9348.226211] Call Trace: [ 9348.228914] [ 9348.231010] ? show_regs.cold.9+0x22/0x2f [ 9348.236710] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9348.240120] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9348.246071] ? watchdog+0x40/0x40 [ 9348.249950] ? hrtimer_interrupt+0x128/0x2c0 [ 9348.254971] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9348.260785] ? apic_timer_interrupt+0xf/0x20 [ 9348.262371] [ 9348.263138] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9348.270926] do_raw_spin_lock+0x177/0x190 [ 9348.276535] _raw_spin_lock+0x21/0x30 [ 9348.280410] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9348.284344] ? do_raw_spin_unlock+0x75/0x190 [ 9348.285514] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9348.289020] ? do_wait_intr+0xf0/0xf0 [ 9348.292239] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9348.293579] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9348.299981] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9348.305090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9348.307533] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9348.310223] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9348.313207] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9348.314719] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9348.317946] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9348.321219] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9348.324652] lmv_intent_open+0x338/0x1150 [lmv] [ 9348.327025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9348.329936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9348.332057] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9348.333853] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9348.335899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9348.339880] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9348.344209] ? do_raw_spin_unlock+0x75/0x190 [ 9348.346993] ? _raw_spin_unlock+0x12/0x30 [ 9348.349173] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9348.355415] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9348.360807] ll_atomic_open+0x36a/0xd70 [lustre] [ 9348.365547] lookup_open+0x356/0xb40 [ 9348.368192] do_last+0x313/0xfc0 [ 9348.372698] ? nd_jump_root+0xe5/0x160 [ 9348.375814] ? path_init+0x437/0x520 [ 9348.378110] path_openat+0xf7/0x500 [ 9348.379164] ? do_raw_spin_unlock+0x75/0x190 [ 9348.382574] do_filp_open+0x99/0x140 [ 9348.386317] ? getname_flags+0x6e/0x330 [ 9348.390019] ? __check_object_size+0xff/0x256 [ 9348.394005] ? do_raw_spin_unlock+0x75/0x190 [ 9348.395952] ? _raw_spin_unlock+0x12/0x30 [ 9348.397650] do_sys_openat2+0x2b4/0x410 [ 9348.406426] do_sys_open+0x73/0xa0 [ 9348.407687] __x64_sys_openat+0x24/0x30 [ 9348.408799] do_syscall_64+0xc1/0x440 [ 9348.414043] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9348.418142] RIP: 0033:0x7f1a63983332 [ 9348.422154] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9348.441219] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9348.447911] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9348.453990] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9348.461904] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9348.468351] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9348.474550] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9353.189322] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7330s! [ 9353.193358] Showing busy workqueues and worker pools: [ 9353.196289] workqueue events: flags=0x0 [ 9353.198680] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9353.206634] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9353.213937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9353.217495] pending: sec_gc_main [ptlrpc] [ 9353.221090] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9353.225596] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9353.232391] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9353.235603] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9353.239556] workqueue events_long: flags=0x0 [ 9353.241192] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9353.243844] pending: lu_objects_flush [obdclass] [ 9353.246581] workqueue events_unbound: flags=0x2 [ 9353.248872] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9353.251455] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9353.255769] workqueue events_power_efficient: flags=0x80 [ 9353.257734] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9353.261147] pending: neigh_periodic_work, do_cache_clean [ 9353.263389] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9353.265853] pending: neigh_periodic_work, check_lifetime [ 9353.267985] workqueue mm_percpu_wq: flags=0x8 [ 9353.269655] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9353.272340] pending: vmstat_update [ 9353.273575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9353.275705] pending: vmstat_update [ 9353.277116] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9353.279663] pending: vmstat_update [ 9353.280962] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9353.283797] pending: vmstat_update [ 9353.285362] workqueue writeback: flags=0x4a [ 9353.286852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9353.289571] pending: wb_workfn, wb_workfn [ 9353.291340] workqueue xprtiod: flags=0xa [ 9353.292647] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9353.295063] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9353.298666] workqueue ipv6_addrconf: flags=0x40008 [ 9353.300881] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9353.304262] pending: addrconf_verify_work [ 9353.305713] workqueue knbd0-recv: flags=0x1a [ 9353.307393] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9353.309632] in-flight: 647:recv_work [ 9353.311265] workqueue ptlrpc_pinger: flags=0x2 [ 9353.312825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9353.315458] pending: ptlrpc_pinger_main [ptlrpc] [ 9353.318272] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9343s workers=2 idle: 652 [ 9364.058022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9364.061656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9364.087331] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9364.095813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9364.103337] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9364.104759] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9364.116154] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9364.121997] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9364.127031] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9364.134453] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9364.136861] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9364.151559] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9364.168765] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9364.176004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9364.178695] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9364.183047] Call Trace: [ 9364.184682] [ 9364.185412] ? show_regs.cold.9+0x22/0x2f [ 9364.187935] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9364.190374] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9364.192309] ? watchdog+0x40/0x40 [ 9364.194323] ? hrtimer_interrupt+0x128/0x2c0 [ 9364.196287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9364.199233] ? apic_timer_interrupt+0xf/0x20 [ 9364.201763] [ 9364.202592] ? native_safe_halt+0x2e/0x50 [ 9364.205603] kvm_wait+0xcd/0xe0 [ 9364.206756] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9364.209591] do_raw_spin_lock+0x177/0x190 [ 9364.211112] _raw_spin_lock+0x21/0x30 [ 9364.213044] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9364.216735] obd_mod_free+0x46/0x140 [ptlrpc] [ 9364.218841] mdc_commit_open+0x91/0xb0 [mdc] [ 9364.220390] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9364.222250] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9364.224585] after_reply+0xe2c/0x1780 [ptlrpc] [ 9364.226816] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9364.229478] ? __next_timer_interrupt+0x160/0x160 [ 9364.231076] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9364.232773] ? do_wait_intr+0xf0/0xf0 [ 9364.234207] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9364.235871] kthread+0x1d1/0x200 [ 9364.236792] ? set_kthread_struct+0x70/0x70 [ 9364.237969] ret_from_fork+0x1f/0x30 [ 9368.056041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9368.057049] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9368.060547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 9368.070730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9368.070828] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9368.070835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9368.070839] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.070865] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9368.070869] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9368.070874] RAX: 0000000000001169 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9368.070877] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9368.070879] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9368.070881] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9368.070883] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9368.070887] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9368.070891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9368.070893] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9368.070901] Call Trace: [ 9368.070906] [ 9368.070912] ? show_regs.cold.9+0x22/0x2f [ 9368.070922] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9368.070929] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9368.070933] ? watchdog+0x40/0x40 [ 9368.070938] ? hrtimer_interrupt+0x128/0x2c0 [ 9368.070942] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9368.070951] ? apic_timer_interrupt+0xf/0x20 [ 9368.070956] [ 9368.070959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.070965] do_raw_spin_lock+0x177/0x190 [ 9368.070969] _raw_spin_lock+0x21/0x30 [ 9368.070977] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9368.078638] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9368.078710] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9368.078718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9368.078721] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.078736] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9368.078740] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9368.078745] RAX: 0000000000001586 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9368.078747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9368.078749] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9368.078751] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9368.078754] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9368.078831] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9368.078835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9368.078837] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9368.078845] Call Trace: [ 9368.078851] [ 9368.078862] ? show_regs.cold.9+0x22/0x2f [ 9368.078873] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9368.078879] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9368.078884] ? watchdog+0x40/0x40 [ 9368.078889] ? hrtimer_interrupt+0x128/0x2c0 [ 9368.078893] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9368.078900] ? apic_timer_interrupt+0xf/0x20 [ 9368.078904] [ 9368.078907] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.078912] do_raw_spin_lock+0x177/0x190 [ 9368.078918] _raw_spin_lock+0x21/0x30 [ 9368.078922] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9368.079565] ? do_raw_spin_unlock+0x75/0x190 [ 9368.079571] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9368.079900] ? do_wait_intr+0xf0/0xf0 [ 9368.079908] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9368.080394] mdc_close+0x53b/0xd70 [mdc] [ 9368.080436] lmv_close+0x18d/0x490 [lmv] [ 9368.080556] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9368.080717] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9368.080893] ll_md_close+0x274/0x800 [lustre] [ 9368.080978] ll_file_release+0x128/0xc50 [lustre] [ 9368.081081] __fput+0x10d/0x440 [ 9368.081089] ____fput+0x12/0x20 [ 9368.081094] task_work_run+0xc6/0x110 [ 9368.081101] exit_to_usermode_loop+0x1dd/0x1f0 [ 9368.081107] do_syscall_64+0x3d6/0x440 [ 9368.081112] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9368.081119] RIP: 0033:0x7f1a63983cd8 [ 9368.081130] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9368.081134] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9368.081142] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9368.081144] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9368.081147] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9368.081149] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9368.081151] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9368.477905] ? do_raw_spin_unlock+0x75/0x190 [ 9368.481690] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9368.483907] ? do_wait_intr+0xf0/0xf0 [ 9368.493351] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9368.500590] mdc_close+0x53b/0xd70 [mdc] [ 9368.506331] lmv_close+0x18d/0x490 [lmv] [ 9368.507435] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9368.514540] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9368.525883] ll_md_close+0x274/0x800 [lustre] [ 9368.527729] ll_file_release+0x128/0xc50 [lustre] [ 9368.531824] __fput+0x10d/0x440 [ 9368.535572] ____fput+0x12/0x20 [ 9368.537546] task_work_run+0xc6/0x110 [ 9368.543429] exit_to_usermode_loop+0x1dd/0x1f0 [ 9368.550269] do_syscall_64+0x3d6/0x440 [ 9368.552861] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9368.561766] RIP: 0033:0x7f1a63983cd8 [ 9368.562891] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9368.577228] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9368.588469] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9368.596365] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9368.601263] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9368.604312] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9368.612379] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9380.053030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9380.056307] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9380.080688] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9380.086024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9380.091131] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9380.093365] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9380.107066] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9380.109586] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9380.114344] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9380.118453] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9380.121763] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9380.126258] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9380.136546] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9380.142572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9380.148587] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9380.150586] Call Trace: [ 9380.151421] [ 9380.153686] ? show_regs.cold.9+0x22/0x2f [ 9380.155637] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9380.157607] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9380.159393] ? watchdog+0x40/0x40 [ 9380.160571] ? hrtimer_interrupt+0x128/0x2c0 [ 9380.165330] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9380.168608] ? apic_timer_interrupt+0xf/0x20 [ 9380.169804] [ 9380.170396] ? native_safe_halt+0x2e/0x50 [ 9380.171469] kvm_wait+0xcd/0xe0 [ 9380.175094] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9380.177209] do_raw_spin_lock+0x177/0x190 [ 9380.179081] _raw_spin_lock+0x21/0x30 [ 9380.180304] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9380.182641] ? do_raw_spin_unlock+0x75/0x190 [ 9380.183810] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9380.185177] ? do_wait_intr+0xf0/0xf0 [ 9380.186283] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9380.187972] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9380.189767] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9380.192293] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9380.194731] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9380.196642] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9380.198348] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9380.199830] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9380.201686] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9380.204173] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9380.206478] lmv_intent_open+0x338/0x1150 [lmv] [ 9380.208424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9380.210174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9380.212169] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9380.213780] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9380.215224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9380.216881] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9380.218487] ? do_raw_spin_unlock+0x75/0x190 [ 9380.220262] ? _raw_spin_unlock+0x12/0x30 [ 9380.221856] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9380.223735] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9380.225641] ll_atomic_open+0x36a/0xd70 [lustre] [ 9380.227168] lookup_open+0x356/0xb40 [ 9380.228381] do_last+0x313/0xfc0 [ 9380.230073] ? nd_jump_root+0xe5/0x160 [ 9380.232435] ? path_init+0x437/0x520 [ 9380.233586] path_openat+0xf7/0x500 [ 9380.234869] ? do_raw_spin_unlock+0x75/0x190 [ 9380.236586] do_filp_open+0x99/0x140 [ 9380.237866] ? getname_flags+0x6e/0x330 [ 9380.239281] ? __check_object_size+0xff/0x256 [ 9380.240811] ? do_raw_spin_unlock+0x75/0x190 [ 9380.242600] ? _raw_spin_unlock+0x12/0x30 [ 9380.243836] do_sys_openat2+0x2b4/0x410 [ 9380.245210] do_sys_open+0x73/0xa0 [ 9380.246300] __x64_sys_openat+0x24/0x30 [ 9380.247551] do_syscall_64+0xc1/0x440 [ 9380.248854] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9380.250864] RIP: 0033:0x7f1a63983332 [ 9380.252119] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9380.260578] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9380.262734] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9380.265450] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9380.269104] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9380.272731] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9380.274918] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9383.392985] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7360s! [ 9383.397694] Showing busy workqueues and worker pools: [ 9383.400291] workqueue events: flags=0x0 [ 9383.401872] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9383.404830] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9383.408954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9383.411877] pending: sec_gc_main [ptlrpc] [ 9383.418197] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9383.421367] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9383.431610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9383.436299] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9383.453325] workqueue events_long: flags=0x0 [ 9383.460869] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9383.468076] pending: lu_objects_flush [obdclass] [ 9383.475125] workqueue events_unbound: flags=0x2 [ 9383.478786] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9383.482251] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9383.488509] workqueue events_power_efficient: flags=0x80 [ 9383.490391] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9383.497977] pending: neigh_periodic_work, do_cache_clean [ 9383.506385] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9383.512302] pending: neigh_periodic_work, check_lifetime [ 9383.515990] workqueue mm_percpu_wq: flags=0x8 [ 9383.518843] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9383.525807] pending: vmstat_update [ 9383.530368] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9383.535388] pending: vmstat_update [ 9383.540932] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9383.544947] pending: vmstat_update [ 9383.547348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9383.551316] pending: vmstat_update [ 9383.553298] workqueue writeback: flags=0x4a [ 9383.555842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9383.561474] pending: wb_workfn, wb_workfn [ 9383.567494] workqueue xprtiod: flags=0xa [ 9383.572856] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9383.577567] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9383.582061] workqueue ipv6_addrconf: flags=0x40008 [ 9383.584537] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9383.591416] pending: addrconf_verify_work [ 9383.595895] workqueue knbd0-recv: flags=0x1a [ 9383.597810] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9383.601312] in-flight: 647:recv_work [ 9383.604296] workqueue ptlrpc_pinger: flags=0x2 [ 9383.607524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9383.611295] pending: ptlrpc_pinger_main [ptlrpc] [ 9383.615511] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9373s workers=2 idle: 652 [ 9392.058922] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 9392.067113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9392.099358] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9392.104467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9392.109155] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9392.112028] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9392.122126] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9392.125931] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9392.128784] RDX: 00000000000006c9 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9392.132350] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9392.136224] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9392.139285] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9392.142398] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9392.145873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9392.148770] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9392.152355] Call Trace: [ 9392.153411] [ 9392.154353] ? show_regs.cold.9+0x22/0x2f [ 9392.155916] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9392.157662] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9392.159332] ? watchdog+0x40/0x40 [ 9392.160414] ? hrtimer_interrupt+0x128/0x2c0 [ 9392.161748] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9392.163154] ? apic_timer_interrupt+0xf/0x20 [ 9392.164417] [ 9392.165134] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9392.166675] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9392.168443] do_raw_spin_lock+0x177/0x190 [ 9392.169667] _raw_spin_lock+0x21/0x30 [ 9392.170989] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9392.173356] obd_mod_free+0x46/0x140 [ptlrpc] [ 9392.175614] mdc_commit_open+0x91/0xb0 [mdc] [ 9392.177395] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9392.179984] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9392.182765] after_reply+0xe2c/0x1780 [ptlrpc] [ 9392.185250] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9392.187678] ? __next_timer_interrupt+0x160/0x160 [ 9392.189775] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9392.191928] ? do_wait_intr+0xf0/0xf0 [ 9392.193446] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9392.196027] kthread+0x1d1/0x200 [ 9392.197332] ? set_kthread_struct+0x70/0x70 [ 9392.199451] ret_from_fork+0x1f/0x30 [ 9396.056041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9396.057064] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9396.065519] Modules linked in: [ 9396.069711] Modules linked in: [ 9396.069715] lustre(O) osp(O) [ 9396.071357] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 9396.072752] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [ 9396.080998] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9396.081051] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9396.081058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9396.081061] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9396.081075] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9396.081079] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9396.081084] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9396.081086] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9396.081088] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9396.081090] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9396.081093] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9396.081096] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9396.081099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9396.081101] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9396.081110] Call Trace: [ 9396.081116] [ 9396.081122] ? show_regs.cold.9+0x22/0x2f [ 9396.081133] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9396.081138] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9396.081143] ? watchdog+0x40/0x40 [ 9396.081147] ? hrtimer_interrupt+0x128/0x2c0 [ 9396.081151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9396.081158] ? apic_timer_interrupt+0xf/0x20 [ 9396.081235] [ 9396.081238] ? native_safe_halt+0x2e/0x50 [ 9396.081242] kvm_wait+0xcd/0xe0 [ 9396.081250] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9396.081263] do_raw_spin_lock+0x177/0x190 [ 9396.081268] _raw_spin_lock+0x21/0x30 [ 9396.081272] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9396.081555] ? do_raw_spin_unlock+0x75/0x190 [ 9396.081561] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9396.081765] ? do_wait_intr+0xf0/0xf0 [ 9396.081772] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9396.082171] mdc_close+0x53b/0xd70 [mdc] [ 9396.082208] lmv_close+0x18d/0x490 [lmv] [ 9396.082237] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9396.082410] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9396.082504] ll_md_close+0x274/0x800 [lustre] [ 9396.082596] ll_file_release+0x128/0xc50 [lustre] [ 9396.082691] __fput+0x10d/0x440 [ 9396.082700] ____fput+0x12/0x20 [ 9396.082704] task_work_run+0xc6/0x110 [ 9396.082712] exit_to_usermode_loop+0x1dd/0x1f0 [ 9396.082718] do_syscall_64+0x3d6/0x440 [ 9396.082722] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9396.082728] RIP: 0033:0x7f1a63983cd8 [ 9396.082735] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9396.082739] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9396.082744] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9396.082746] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9396.082749] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9396.082751] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9396.082754] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9396.241189] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9396.254423] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9396.260293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9396.265624] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9396.268673] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9396.287303] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9396.289405] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9396.292269] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9396.295521] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9396.302100] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9396.305904] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9396.308507] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9396.312163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9396.316410] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9396.319280] Call Trace: [ 9396.320446] [ 9396.321379] ? show_regs.cold.9+0x22/0x2f [ 9396.323430] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9396.326234] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9396.327724] ? watchdog+0x40/0x40 [ 9396.333138] ? hrtimer_interrupt+0x128/0x2c0 [ 9396.337404] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9396.339558] ? apic_timer_interrupt+0xf/0x20 [ 9396.341757] [ 9396.342717] ? native_safe_halt+0x2e/0x50 [ 9396.348396] kvm_wait+0xcd/0xe0 [ 9396.349698] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9396.352427] do_raw_spin_lock+0x177/0x190 [ 9396.354020] _raw_spin_lock+0x21/0x30 [ 9396.356362] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9396.359123] ? do_raw_spin_unlock+0x75/0x190 [ 9396.360519] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9396.367357] ? do_wait_intr+0xf0/0xf0 [ 9396.368752] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9396.371604] mdc_close+0x53b/0xd70 [mdc] [ 9396.373557] lmv_close+0x18d/0x490 [lmv] [ 9396.375446] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9396.378454] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9396.380539] ll_md_close+0x274/0x800 [lustre] [ 9396.382703] ll_file_release+0x128/0xc50 [lustre] [ 9396.384615] __fput+0x10d/0x440 [ 9396.386262] ____fput+0x12/0x20 [ 9396.387977] task_work_run+0xc6/0x110 [ 9396.389622] exit_to_usermode_loop+0x1dd/0x1f0 [ 9396.391596] do_syscall_64+0x3d6/0x440 [ 9396.393508] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9396.396028] RIP: 0033:0x7f1a63983cd8 [ 9396.397571] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9396.409657] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9396.412966] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9396.416374] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9396.419586] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9396.423183] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9396.427368] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9408.057603] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9408.065245] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9408.098595] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9408.106897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9408.111597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9408.116054] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9408.133428] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9408.140641] RAX: 000000000000700d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9408.148347] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9408.152087] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9408.155359] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9408.159340] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9408.163269] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9408.167808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9408.171301] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9408.175315] Call Trace: [ 9408.176591] [ 9408.177628] ? show_regs.cold.9+0x22/0x2f [ 9408.182855] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9408.188962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9408.190616] ? watchdog+0x40/0x40 [ 9408.194268] ? hrtimer_interrupt+0x128/0x2c0 [ 9408.197684] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9408.200912] ? apic_timer_interrupt+0xf/0x20 [ 9408.203657] [ 9408.205155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9408.207513] do_raw_spin_lock+0x177/0x190 [ 9408.209545] _raw_spin_lock+0x21/0x30 [ 9408.212067] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9408.215802] ? do_raw_spin_unlock+0x75/0x190 [ 9408.217768] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9408.225031] ? do_wait_intr+0xf0/0xf0 [ 9408.229026] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9408.233060] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9408.235504] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9408.238343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9408.240795] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9408.243109] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9408.251027] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9408.256063] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9408.259962] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9408.265997] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9408.272612] lmv_intent_open+0x338/0x1150 [lmv] [ 9408.276879] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9408.282140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9408.288244] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9408.293055] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9408.303075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9408.312851] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9408.320688] ? do_raw_spin_unlock+0x75/0x190 [ 9408.324494] ? _raw_spin_unlock+0x12/0x30 [ 9408.326845] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9408.331640] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9408.340599] ll_atomic_open+0x36a/0xd70 [lustre] [ 9408.346003] lookup_open+0x356/0xb40 [ 9408.348258] do_last+0x313/0xfc0 [ 9408.349402] ? nd_jump_root+0xe5/0x160 [ 9408.354861] ? path_init+0x437/0x520 [ 9408.357230] path_openat+0xf7/0x500 [ 9408.359179] ? do_raw_spin_unlock+0x75/0x190 [ 9408.362161] do_filp_open+0x99/0x140 [ 9408.364278] ? getname_flags+0x6e/0x330 [ 9408.366513] ? __check_object_size+0xff/0x256 [ 9408.368646] ? do_raw_spin_unlock+0x75/0x190 [ 9408.370488] ? _raw_spin_unlock+0x12/0x30 [ 9408.372359] do_sys_openat2+0x2b4/0x410 [ 9408.374321] do_sys_open+0x73/0xa0 [ 9408.375302] __x64_sys_openat+0x24/0x30 [ 9408.377321] do_syscall_64+0xc1/0x440 [ 9408.379572] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9408.382901] RIP: 0033:0x7f1a63983332 [ 9408.384835] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9408.395269] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9408.404077] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9408.411862] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9408.418737] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9408.423641] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9408.427111] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9414.112048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7391s! [ 9414.117663] Showing busy workqueues and worker pools: [ 9414.120518] workqueue events: flags=0x0 [ 9414.122728] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9414.127330] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9414.132273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9414.136297] pending: sec_gc_main [ptlrpc] [ 9414.139956] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9414.149815] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9414.155010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9414.158511] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9414.162865] workqueue events_long: flags=0x0 [ 9414.167293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9414.170201] pending: lu_objects_flush [obdclass] [ 9414.172558] workqueue events_unbound: flags=0x2 [ 9414.176327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9414.181047] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9414.188408] workqueue events_power_efficient: flags=0x80 [ 9414.190563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9414.193501] pending: neigh_periodic_work, do_cache_clean [ 9414.195978] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9414.198758] pending: neigh_periodic_work, check_lifetime [ 9414.200800] workqueue mm_percpu_wq: flags=0x8 [ 9414.202404] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9414.204906] pending: vmstat_update [ 9414.206431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9414.208697] pending: vmstat_update [ 9414.210032] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9414.212775] pending: vmstat_update [ 9414.214179] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9414.217376] pending: vmstat_update [ 9414.218946] workqueue writeback: flags=0x4a [ 9414.220884] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9414.224544] pending: wb_workfn, wb_workfn [ 9414.226451] workqueue xprtiod: flags=0xa [ 9414.227760] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9414.229737] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9414.232986] workqueue ipv6_addrconf: flags=0x40008 [ 9414.235067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9414.238554] pending: addrconf_verify_work [ 9414.242061] workqueue knbd0-recv: flags=0x1a [ 9414.244247] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9414.246428] in-flight: 647:recv_work [ 9414.249676] workqueue ptlrpc_pinger: flags=0x2 [ 9414.254825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9414.257863] pending: ptlrpc_pinger_main [ptlrpc] [ 9414.265899] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9404s workers=2 idle: 652 [ 9420.058484] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 9420.076042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9420.126051] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9420.139254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9420.151343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9420.157061] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9420.180399] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9420.187323] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9420.194284] RDX: 0000000000002095 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9420.200804] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9420.205771] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9420.213297] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9420.218443] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9420.225677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9420.228390] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9420.232400] Call Trace: [ 9420.233308] [ 9420.234143] ? show_regs.cold.9+0x22/0x2f [ 9420.240301] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9420.247987] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9420.252211] ? watchdog+0x40/0x40 [ 9420.254317] ? hrtimer_interrupt+0x128/0x2c0 [ 9420.256407] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9420.264320] ? apic_timer_interrupt+0xf/0x20 [ 9420.268359] [ 9420.270846] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9420.274439] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9420.277953] do_raw_spin_lock+0x177/0x190 [ 9420.280333] _raw_spin_lock+0x21/0x30 [ 9420.282474] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9420.288019] obd_mod_free+0x46/0x140 [ptlrpc] [ 9420.291350] mdc_commit_open+0x91/0xb0 [mdc] [ 9420.293411] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9420.296859] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9420.300253] after_reply+0xe2c/0x1780 [ptlrpc] [ 9420.304392] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9420.317941] ? __next_timer_interrupt+0x160/0x160 [ 9420.320270] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9420.328274] ? do_wait_intr+0xf0/0xf0 [ 9420.331348] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9420.337336] kthread+0x1d1/0x200 [ 9420.339852] ? set_kthread_struct+0x70/0x70 [ 9420.342446] ret_from_fork+0x1f/0x30 [ 9424.056085] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9424.057019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9424.059936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9424.059994] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9424.060001] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9424.064222] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9424.064299] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9424.064563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9424.064566] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9424.064580] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9424.064584] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9424.064589] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9424.064591] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9424.064593] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9424.064595] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9424.064598] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9424.064601] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9424.064604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9424.064606] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9424.064616] Call Trace: [ 9424.064622] [ 9424.064628] ? show_regs.cold.9+0x22/0x2f [ 9424.064638] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9424.064644] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9424.064649] ? watchdog+0x40/0x40 [ 9424.064726] ? hrtimer_interrupt+0x128/0x2c0 [ 9424.064730] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9424.064737] ? apic_timer_interrupt+0xf/0x20 [ 9424.064742] [ 9424.064745] ? native_safe_halt+0x2e/0x50 [ 9424.064749] kvm_wait+0xcd/0xe0 [ 9424.064757] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9424.064766] do_raw_spin_lock+0x177/0x190 [ 9424.064770] _raw_spin_lock+0x21/0x30 [ 9424.064780] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9424.065135] ? do_raw_spin_unlock+0x75/0x190 [ 9424.065142] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9424.065693] ? do_wait_intr+0xf0/0xf0 [ 9424.065703] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9424.096695] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9424.096719] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9424.096723] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9424.096729] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9424.101283] mdc_close+0x53b/0xd70 [mdc] [ 9424.101334] lmv_close+0x18d/0x490 [lmv] [ 9424.101364] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9424.101476] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9424.101628] ll_md_close+0x274/0x800 [lustre] [ 9424.101712] ll_file_release+0x128/0xc50 [lustre] [ 9424.101796] __fput+0x10d/0x440 [ 9424.101806] ____fput+0x12/0x20 [ 9424.101810] task_work_run+0xc6/0x110 [ 9424.101818] exit_to_usermode_loop+0x1dd/0x1f0 [ 9424.101825] do_syscall_64+0x3d6/0x440 [ 9424.101829] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9424.101838] RIP: 0033:0x7f1a63983cd8 [ 9424.101847] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9424.101851] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9424.101856] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9424.101859] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9424.101861] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9424.101863] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9424.101866] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9424.353992] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9424.358322] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9424.362559] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9424.367782] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9424.371163] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9424.378552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9424.381944] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9424.385229] Call Trace: [ 9424.386180] [ 9424.387141] ? show_regs.cold.9+0x22/0x2f [ 9424.389230] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9424.392428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9424.395138] ? watchdog+0x40/0x40 [ 9424.397222] ? hrtimer_interrupt+0x128/0x2c0 [ 9424.399645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9424.402013] ? apic_timer_interrupt+0xf/0x20 [ 9424.405380] [ 9424.406370] ? native_safe_halt+0x2e/0x50 [ 9424.408349] kvm_wait+0xcd/0xe0 [ 9424.410290] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9424.412291] do_raw_spin_lock+0x177/0x190 [ 9424.414395] _raw_spin_lock+0x21/0x30 [ 9424.416181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9424.418779] ? do_raw_spin_unlock+0x75/0x190 [ 9424.420929] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9424.423574] ? do_wait_intr+0xf0/0xf0 [ 9424.424988] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9424.427329] mdc_close+0x53b/0xd70 [mdc] [ 9424.429310] lmv_close+0x18d/0x490 [lmv] [ 9424.431092] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9424.433854] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9424.436260] ll_md_close+0x274/0x800 [lustre] [ 9424.438339] ll_file_release+0x128/0xc50 [lustre] [ 9424.441081] __fput+0x10d/0x440 [ 9424.442750] ____fput+0x12/0x20 [ 9424.444079] task_work_run+0xc6/0x110 [ 9424.445859] exit_to_usermode_loop+0x1dd/0x1f0 [ 9424.448031] do_syscall_64+0x3d6/0x440 [ 9424.449374] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9424.451687] RIP: 0033:0x7f1a63983cd8 [ 9424.453219] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9424.461599] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9424.465195] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9424.468276] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9424.471897] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9424.475718] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9424.479102] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9440.054056] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9440.065915] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9440.130505] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9440.143098] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9440.153993] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9440.163093] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9440.184777] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9440.192845] RAX: 0000000000000290 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9440.208986] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9440.227337] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9440.247396] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9440.262233] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9440.275558] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9440.289225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9440.300206] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9440.316537] Call Trace: [ 9440.321220] [ 9440.326758] ? show_regs.cold.9+0x22/0x2f [ 9440.329245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9440.333305] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9440.338842] ? watchdog+0x40/0x40 [ 9440.342093] ? hrtimer_interrupt+0x128/0x2c0 [ 9440.347239] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9440.351348] ? apic_timer_interrupt+0xf/0x20 [ 9440.354952] [ 9440.357441] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9440.363392] do_raw_spin_lock+0x177/0x190 [ 9440.367975] _raw_spin_lock+0x21/0x30 [ 9440.374136] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9440.382455] ? do_raw_spin_unlock+0x75/0x190 [ 9440.391423] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9440.403744] ? do_wait_intr+0xf0/0xf0 [ 9440.409585] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9440.413565] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9440.416836] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9440.424181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9440.429226] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9440.444468] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9440.459168] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9440.470052] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9440.474659] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9440.478150] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9440.485289] lmv_intent_open+0x338/0x1150 [lmv] [ 9440.498169] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9440.510710] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9440.518226] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9440.525328] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9440.535746] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9440.543489] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9440.547051] ? do_raw_spin_unlock+0x75/0x190 [ 9440.553090] ? _raw_spin_unlock+0x12/0x30 [ 9440.558266] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9440.561334] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9440.563247] ll_atomic_open+0x36a/0xd70 [lustre] [ 9440.566454] lookup_open+0x356/0xb40 [ 9440.569963] do_last+0x313/0xfc0 [ 9440.573865] ? nd_jump_root+0xe5/0x160 [ 9440.581131] ? path_init+0x437/0x520 [ 9440.591141] path_openat+0xf7/0x500 [ 9440.597118] ? do_raw_spin_unlock+0x75/0x190 [ 9440.602842] do_filp_open+0x99/0x140 [ 9440.604128] ? getname_flags+0x6e/0x330 [ 9440.612071] ? __check_object_size+0xff/0x256 [ 9440.624444] ? do_raw_spin_unlock+0x75/0x190 [ 9440.631976] ? _raw_spin_unlock+0x12/0x30 [ 9440.639130] do_sys_openat2+0x2b4/0x410 [ 9440.640558] do_sys_open+0x73/0xa0 [ 9440.648124] __x64_sys_openat+0x24/0x30 [ 9440.649599] do_syscall_64+0xc1/0x440 [ 9440.650975] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9440.667173] RIP: 0033:0x7f1a63983332 [ 9440.678585] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9440.719461] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9440.735301] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9440.747467] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9440.757797] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9440.783467] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9440.798707] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9444.325512] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7421s! [ 9444.335752] Showing busy workqueues and worker pools: [ 9444.347089] workqueue events: flags=0x0 [ 9444.351091] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9444.358875] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9444.364452] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9444.367887] pending: sec_gc_main [ptlrpc] [ 9444.371127] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9444.379186] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9444.387080] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9444.399018] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9444.419408] workqueue events_long: flags=0x0 [ 9444.427367] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9444.437234] pending: lu_objects_flush [obdclass] [ 9444.442876] workqueue events_unbound: flags=0x2 [ 9444.453014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9444.457785] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9444.482194] workqueue events_power_efficient: flags=0x80 [ 9444.490736] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9444.497653] pending: neigh_periodic_work, do_cache_clean [ 9444.506696] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9444.513187] pending: neigh_periodic_work, check_lifetime [ 9444.524599] workqueue mm_percpu_wq: flags=0x8 [ 9444.532364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9444.547132] pending: vmstat_update [ 9444.548484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9444.561593] pending: vmstat_update [ 9444.568079] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9444.583791] pending: vmstat_update [ 9444.588644] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9444.600280] pending: vmstat_update [ 9444.604352] workqueue writeback: flags=0x4a [ 9444.608060] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9444.615897] pending: wb_workfn, wb_workfn [ 9444.619228] workqueue xprtiod: flags=0xa [ 9444.623955] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9444.635431] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9444.640966] workqueue ipv6_addrconf: flags=0x40008 [ 9444.649281] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9444.660378] pending: addrconf_verify_work [ 9444.667629] workqueue knbd0-recv: flags=0x1a [ 9444.669237] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9444.683711] in-flight: 647:recv_work [ 9444.689988] workqueue ptlrpc_pinger: flags=0x2 [ 9444.697057] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9444.703726] pending: ptlrpc_pinger_main [ptlrpc] [ 9444.708721] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9434s workers=2 idle: 652 [ 9448.058628] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 9448.069228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9448.145152] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9448.161269] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9448.169544] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9448.172946] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9448.203256] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9448.215866] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9448.222236] RDX: 00000000000060cb RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9448.229098] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9448.237174] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9448.242086] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9448.245477] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9448.249207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9448.251624] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9448.256242] Call Trace: [ 9448.257628] [ 9448.259087] ? show_regs.cold.9+0x22/0x2f [ 9448.260599] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9448.263781] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9448.266256] ? watchdog+0x40/0x40 [ 9448.267724] ? hrtimer_interrupt+0x128/0x2c0 [ 9448.271558] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9448.273905] ? apic_timer_interrupt+0xf/0x20 [ 9448.277224] [ 9448.278384] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9448.283714] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9448.287035] do_raw_spin_lock+0x177/0x190 [ 9448.289115] _raw_spin_lock+0x21/0x30 [ 9448.292483] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9448.295209] obd_mod_free+0x46/0x140 [ptlrpc] [ 9448.297932] mdc_commit_open+0x91/0xb0 [mdc] [ 9448.301810] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9448.310498] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9448.317506] after_reply+0xe2c/0x1780 [ptlrpc] [ 9448.321685] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9448.329821] ? __next_timer_interrupt+0x160/0x160 [ 9448.335190] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9448.341803] ? do_wait_intr+0xf0/0xf0 [ 9448.350214] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9448.361827] kthread+0x1d1/0x200 [ 9448.364066] ? set_kthread_struct+0x70/0x70 [ 9448.369045] ret_from_fork+0x1f/0x30 [ 9452.057026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9452.061579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9452.064650] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9452.092966] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9452.092982] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9452.092985] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9452.092999] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9452.093003] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9452.093008] RAX: 0000000000006885 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9452.093010] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9452.093012] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9452.093014] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9452.093016] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9452.093019] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9452.093021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9452.093023] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9452.093031] Call Trace: [ 9452.093036] [ 9452.093042] ? show_regs.cold.9+0x22/0x2f [ 9452.093052] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9452.093059] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9452.093063] ? watchdog+0x40/0x40 [ 9452.093067] ? hrtimer_interrupt+0x128/0x2c0 [ 9452.093071] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9452.093078] ? apic_timer_interrupt+0xf/0x20 [ 9452.093082] [ 9452.093085] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9452.093091] do_raw_spin_lock+0x177/0x190 [ 9452.093096] _raw_spin_lock+0x21/0x30 [ 9452.093100] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9452.093335] ? do_raw_spin_unlock+0x75/0x190 [ 9452.093341] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9452.093541] ? do_wait_intr+0xf0/0xf0 [ 9452.093549] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9452.093741] mdc_close+0x53b/0xd70 [mdc] [ 9452.094057] lmv_close+0x18d/0x490 [lmv] [ 9452.094090] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9452.115116] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9452.115196] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9452.115203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9452.115207] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9452.115232] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9452.115237] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9452.115241] RAX: 000000000000780c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9452.115243] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9452.130941] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9452.131064] ll_md_close+0x274/0x800 [lustre] [ 9452.131146] ll_file_release+0x128/0xc50 [lustre] [ 9452.131815] __fput+0x10d/0x440 [ 9452.131826] ____fput+0x12/0x20 [ 9452.131836] task_work_run+0xc6/0x110 [ 9452.131844] exit_to_usermode_loop+0x1dd/0x1f0 [ 9452.131852] do_syscall_64+0x3d6/0x440 [ 9452.131857] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9452.131866] RIP: 0033:0x7f1a63983cd8 [ 9452.131874] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9452.131879] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9452.131884] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9452.131886] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9452.131888] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9452.131890] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9452.131893] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9452.618079] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9452.621101] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9452.624391] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9452.627598] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9452.631009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9452.633397] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9452.636536] Call Trace: [ 9452.637772] [ 9452.638788] ? show_regs.cold.9+0x22/0x2f [ 9452.640483] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9452.642717] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9452.644717] ? watchdog+0x40/0x40 [ 9452.646568] ? hrtimer_interrupt+0x128/0x2c0 [ 9452.649517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9452.651538] ? apic_timer_interrupt+0xf/0x20 [ 9452.653723] [ 9452.654825] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9452.657767] do_raw_spin_lock+0x177/0x190 [ 9452.660275] _raw_spin_lock+0x21/0x30 [ 9452.662274] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9452.665656] ? do_raw_spin_unlock+0x75/0x190 [ 9452.667607] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9452.670298] ? do_wait_intr+0xf0/0xf0 [ 9452.672166] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9452.675283] mdc_close+0x53b/0xd70 [mdc] [ 9452.677356] lmv_close+0x18d/0x490 [lmv] [ 9452.679342] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9452.682958] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9452.685209] ll_md_close+0x274/0x800 [lustre] [ 9452.690978] ll_file_release+0x128/0xc50 [lustre] [ 9452.696494] __fput+0x10d/0x440 [ 9452.701733] ____fput+0x12/0x20 [ 9452.704475] task_work_run+0xc6/0x110 [ 9452.707665] exit_to_usermode_loop+0x1dd/0x1f0 [ 9452.709965] do_syscall_64+0x3d6/0x440 [ 9452.713488] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9452.715747] RIP: 0033:0x7f1a63983cd8 [ 9452.718736] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9452.737011] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9452.740031] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9452.752207] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9452.769225] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9452.784545] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9452.793947] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9462.600174] rcu: INFO: rcu_sched self-detected stall on CPU [ 9462.610135] rcu: 3-...!: (4700750 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 9462.629878] (t=7440223 jiffies g=269633 q=2634) [ 9462.639245] rcu: rcu_sched kthread starved for 7439878 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 9462.665447] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9462.682562] rcu: RCU grace-period kthread stack dump: [ 9462.689960] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9462.707902] Call Trace: [ 9462.708788] __schedule+0x351/0xcb0 [ 9462.716067] schedule+0xc0/0x180 [ 9462.723432] schedule_timeout+0xb4/0x190 [ 9462.728973] ? do_raw_spin_unlock+0x75/0x190 [ 9462.729973] ? __next_timer_interrupt+0x160/0x160 [ 9462.749176] ? prepare_to_swait_event+0xf3/0x140 [ 9462.750562] rcu_gp_kthread+0x8dd/0x15b0 [ 9462.751533] ? force_qs_rnp+0x2c0/0x2c0 [ 9462.765865] kthread+0x1d1/0x200 [ 9462.766670] ? set_kthread_struct+0x70/0x70 [ 9462.767633] ret_from_fork+0x1f/0x30 [ 9462.771357] rcu: Stack dump where RCU GP kthread last ran: [ 9462.774536] Sending NMI from CPU 3 to CPUs 0: [ 9462.783799] NMI backtrace for cpu 0 [ 9462.783805] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9462.783807] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9462.783809] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9462.783810] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9462.783812] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 [ 9462.783814] RAX: 0000000000003ebd RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9462.783816] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9462.783817] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9462.783818] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9462.783819] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9462.783821] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9462.783822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9462.783823] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9462.783824] Call Trace: [ 9462.783825] [ 9462.783825] ? show_regs.cold.9+0x22/0x2f [ 9462.783826] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9462.783828] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9462.783829] ? nmi_handle+0xbd/0x210 [ 9462.783829] ? default_do_nmi+0x4b/0x1b0 [ 9462.783830] ? do_nmi+0x280/0x380 [ 9462.783832] ? end_repeat_nmi+0x16/0x1f [ 9462.783833] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9462.783834] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9462.783835] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9462.783836] [ 9462.783837] do_raw_spin_lock+0x177/0x190 [ 9462.783838] _raw_spin_lock+0x21/0x30 [ 9462.783840] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9462.783840] ? do_raw_spin_unlock+0x75/0x190 [ 9462.783841] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9462.783842] ? do_wait_intr+0xf0/0xf0 [ 9462.783843] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9462.783844] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9462.783845] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9462.783847] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9462.783848] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9462.783849] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9462.783850] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9462.783851] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9462.783852] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9462.783853] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9462.783854] lmv_intent_open+0x338/0x1150 [lmv] [ 9462.783855] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9462.783856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9462.783857] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9462.783858] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9462.783859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9462.783861] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9462.783862] ? do_raw_spin_unlock+0x75/0x190 [ 9462.783863] ? _raw_spin_unlock+0x12/0x30 [ 9462.783864] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9462.783866] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9462.783867] ll_atomic_open+0x36a/0xd70 [lustre] [ 9462.783868] lookup_open+0x356/0xb40 [ 9462.783869] do_last+0x313/0xfc0 [ 9462.783870] ? nd_jump_root+0xe5/0x160 [ 9462.783871] ? path_init+0x437/0x520 [ 9462.783872] path_openat+0xf7/0x500 [ 9462.783873] ? do_raw_spin_unlock+0x75/0x190 [ 9462.783874] do_filp_open+0x99/0x140 [ 9462.783875] ? getname_flags+0x6e/0x330 [ 9462.783876] ? __check_object_size+0xff/0x256 [ 9462.783877] ? do_raw_spin_unlock+0x75/0x190 [ 9462.783931] ? _raw_spin_unlock+0x12/0x30 [ 9462.783932] do_sys_openat2+0x2b4/0x410 [ 9462.783933] do_sys_open+0x73/0xa0 [ 9462.783933] __x64_sys_openat+0x24/0x30 [ 9462.783934] do_syscall_64+0xc1/0x440 [ 9462.783935] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9462.783936] RIP: 0033:0x7f1a63983332 [ 9462.783937] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9462.783940] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9462.783942] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9462.783944] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9462.783945] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9462.783947] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9462.783948] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9462.787732] NMI backtrace for cpu 3 [ 9463.066714] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9463.070297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9463.073274] Call Trace: [ 9463.074230] [ 9463.074957] ? dump_stack+0xbb/0x10e [ 9463.076372] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9463.078390] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9463.080326] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9463.082382] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9463.084524] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9463.086633] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9463.088439] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9463.090510] ? update_process_times+0x6d/0xc0 [ 9463.092248] ? tick_sched_handle.isra.18+0x27/0x90 [ 9463.094045] ? tick_sched_timer+0x53/0xb0 [ 9463.095616] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9463.097338] ? tick_sched_handle.isra.18+0x90/0x90 [ 9463.099212] ? hrtimer_interrupt+0x128/0x2c0 [ 9463.101562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9463.102816] ? apic_timer_interrupt+0xf/0x20 [ 9463.104169] [ 9463.104773] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9463.106513] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9463.108056] ? do_raw_spin_lock+0x177/0x190 [ 9463.109129] ? _raw_spin_lock+0x21/0x30 [ 9463.112569] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9463.115381] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9463.117836] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9463.123171] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9463.124751] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9463.127005] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9463.129869] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9463.132078] ? __next_timer_interrupt+0x160/0x160 [ 9463.134005] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9463.136051] ? do_wait_intr+0xf0/0xf0 [ 9463.137615] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9463.140008] ? kthread+0x1d1/0x200 [ 9463.141304] ? set_kthread_struct+0x70/0x70 [ 9463.142966] ? ret_from_fork+0x1f/0x30 [ 9475.045607] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7452s! [ 9475.053828] Showing busy workqueues and worker pools: [ 9475.060954] workqueue events: flags=0x0 [ 9475.067670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9475.073050] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9475.079233] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9475.087238] pending: sec_gc_main [ptlrpc] [ 9475.090698] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9475.093883] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9475.100858] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9475.103282] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9475.109760] workqueue events_long: flags=0x0 [ 9475.111122] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9475.124273] pending: lu_objects_flush [obdclass] [ 9475.134133] workqueue events_unbound: flags=0x2 [ 9475.137720] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9475.139452] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9475.149104] workqueue events_power_efficient: flags=0x80 [ 9475.153725] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9475.156277] pending: neigh_periodic_work, do_cache_clean [ 9475.159165] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9475.160987] pending: neigh_periodic_work, check_lifetime [ 9475.163853] workqueue mm_percpu_wq: flags=0x8 [ 9475.165883] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9475.169542] pending: vmstat_update [ 9475.173394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9475.182843] pending: vmstat_update [ 9475.189429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9475.201413] pending: vmstat_update [ 9475.207009] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9475.224918] pending: vmstat_update [ 9475.227276] workqueue writeback: flags=0x4a [ 9475.236125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9475.248167] pending: wb_workfn, wb_workfn [ 9475.251865] workqueue xprtiod: flags=0xa [ 9475.252978] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9475.258884] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9475.264983] workqueue ipv6_addrconf: flags=0x40008 [ 9475.266266] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9475.267940] pending: addrconf_verify_work [ 9475.283884] workqueue knbd0-recv: flags=0x1a [ 9475.286927] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9475.298821] in-flight: 647:recv_work [ 9475.304605] workqueue ptlrpc_pinger: flags=0x2 [ 9475.306010] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9475.321324] pending: ptlrpc_pinger_main [ptlrpc] [ 9475.325161] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9465s workers=2 idle: 652 [ 9480.056033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9480.057033] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9480.062681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9480.062764] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9480.062771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9480.062774] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9480.062788] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9480.062792] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9480.062797] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9480.062799] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9480.062802] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9480.062804] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9480.062806] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9480.062810] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9480.062813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9480.062815] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9480.062825] Call Trace: [ 9480.062831] [ 9480.062837] ? show_regs.cold.9+0x22/0x2f [ 9480.062847] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9480.062855] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9480.062860] ? watchdog+0x40/0x40 [ 9480.062865] ? hrtimer_interrupt+0x128/0x2c0 [ 9480.062870] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9480.062876] ? apic_timer_interrupt+0xf/0x20 [ 9480.062880] [ 9480.062883] ? native_safe_halt+0x2e/0x50 [ 9480.062887] kvm_wait+0xcd/0xe0 [ 9480.062895] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9480.062903] do_raw_spin_lock+0x177/0x190 [ 9480.062907] _raw_spin_lock+0x21/0x30 [ 9480.062912] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9480.063352] ? do_raw_spin_unlock+0x75/0x190 [ 9480.063358] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9480.063826] ? do_wait_intr+0xf0/0xf0 [ 9480.063833] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9480.064152] mdc_close+0x53b/0xd70 [mdc] [ 9480.064190] lmv_close+0x18d/0x490 [lmv] [ 9480.064287] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9480.067293] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9480.075576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9480.075674] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9480.075681] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9480.075687] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9480.075705] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9480.075712] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9480.075716] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9480.075719] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9480.075720] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9480.075726] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9480.075728] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9480.075731] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9480.075734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9480.075739] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9480.075747] Call Trace: [ 9480.075756] [ 9480.075761] ? show_regs.cold.9+0x22/0x2f [ 9480.075775] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9480.075780] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9480.075784] ? watchdog+0x40/0x40 [ 9480.075792] ? hrtimer_interrupt+0x128/0x2c0 [ 9480.075796] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9480.075803] ? apic_timer_interrupt+0xf/0x20 [ 9480.075811] [ 9480.075814] ? native_safe_halt+0x2e/0x50 [ 9480.075820] kvm_wait+0xcd/0xe0 [ 9480.075840] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9480.075850] do_raw_spin_lock+0x177/0x190 [ 9480.075858] _raw_spin_lock+0x21/0x30 [ 9480.075862] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9480.076388] ? do_raw_spin_unlock+0x75/0x190 [ 9480.076398] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9480.076718] ? do_wait_intr+0xf0/0xf0 [ 9480.076727] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9480.076922] mdc_close+0x53b/0xd70 [mdc] [ 9480.076959] lmv_close+0x18d/0x490 [lmv] [ 9480.077118] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9480.077228] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9480.077308] ll_md_close+0x274/0x800 [lustre] [ 9480.077815] ll_file_release+0x128/0xc50 [lustre] [ 9480.077896] __fput+0x10d/0x440 [ 9480.077906] ____fput+0x12/0x20 [ 9480.077910] task_work_run+0xc6/0x110 [ 9480.077918] exit_to_usermode_loop+0x1dd/0x1f0 [ 9480.077927] do_syscall_64+0x3d6/0x440 [ 9480.077931] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9480.077940] RIP: 0033:0x7f1a63983cd8 [ 9480.077948] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9480.077951] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9480.077956] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9480.077958] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9480.077960] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9480.077962] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9480.077964] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9480.420096] ll_md_close+0x274/0x800 [lustre] [ 9480.423006] ll_file_release+0x128/0xc50 [lustre] [ 9480.425857] __fput+0x10d/0x440 [ 9480.427760] ____fput+0x12/0x20 [ 9480.429509] task_work_run+0xc6/0x110 [ 9480.431495] exit_to_usermode_loop+0x1dd/0x1f0 [ 9480.433820] do_syscall_64+0x3d6/0x440 [ 9480.435678] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9480.438607] RIP: 0033:0x7f1a63983cd8 [ 9480.440557] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9480.451335] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9480.454782] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9480.458949] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9480.463370] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9480.466923] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9480.469885] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9488.061618] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9488.070113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9488.132006] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9488.143623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9488.152730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9488.157753] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9488.181698] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9488.193947] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9488.201676] RDX: 000000000000185a RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9488.210272] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9488.221565] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9488.229103] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9488.237698] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9488.242190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9488.247200] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9488.255614] Call Trace: [ 9488.256443] [ 9488.257174] ? show_regs.cold.9+0x22/0x2f [ 9488.263206] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9488.264837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9488.270496] ? watchdog+0x40/0x40 [ 9488.275638] ? hrtimer_interrupt+0x128/0x2c0 [ 9488.280157] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9488.283662] ? apic_timer_interrupt+0xf/0x20 [ 9488.286541] [ 9488.287676] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9488.293343] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9488.304372] do_raw_spin_lock+0x177/0x190 [ 9488.309542] _raw_spin_lock+0x21/0x30 [ 9488.314331] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9488.323556] obd_mod_free+0x46/0x140 [ptlrpc] [ 9488.331208] mdc_commit_open+0x91/0xb0 [mdc] [ 9488.339112] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9488.342678] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9488.356623] after_reply+0xe2c/0x1780 [ptlrpc] [ 9488.363316] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9488.372374] ? __next_timer_interrupt+0x160/0x160 [ 9488.380067] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9488.385822] ? do_wait_intr+0xf0/0xf0 [ 9488.387024] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9488.402399] kthread+0x1d1/0x200 [ 9488.403646] ? set_kthread_struct+0x70/0x70 [ 9488.411422] ret_from_fork+0x1f/0x30 [ 9500.056069] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9500.068164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9500.110630] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9500.123984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9500.139760] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9500.145356] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9500.170429] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9500.174092] RAX: 0000000000006d82 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9500.177294] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9500.180238] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9500.183306] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9500.186460] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9500.189838] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9500.193659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9500.196237] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9500.200372] Call Trace: [ 9500.201932] [ 9500.202947] ? show_regs.cold.9+0x22/0x2f [ 9500.204754] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9500.207692] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9500.209894] ? watchdog+0x40/0x40 [ 9500.211031] ? hrtimer_interrupt+0x128/0x2c0 [ 9500.212729] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9500.218321] ? apic_timer_interrupt+0xf/0x20 [ 9500.220422] [ 9500.221917] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9500.229289] do_raw_spin_lock+0x177/0x190 [ 9500.238160] _raw_spin_lock+0x21/0x30 [ 9500.242769] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9500.255449] ? do_raw_spin_unlock+0x75/0x190 [ 9500.257026] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9500.274425] ? do_wait_intr+0xf0/0xf0 [ 9500.275754] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9500.291705] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9500.293719] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9500.313988] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9500.320917] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9500.326754] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9500.331903] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9500.336238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9500.338047] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9500.347004] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9500.351225] lmv_intent_open+0x338/0x1150 [lmv] [ 9500.353301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9500.359160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9500.364468] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9500.371051] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9500.376158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9500.382124] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9500.389427] ? do_raw_spin_unlock+0x75/0x190 [ 9500.398691] ? _raw_spin_unlock+0x12/0x30 [ 9500.402962] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9500.406760] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9500.413497] ll_atomic_open+0x36a/0xd70 [lustre] [ 9500.420277] lookup_open+0x356/0xb40 [ 9500.424030] do_last+0x313/0xfc0 [ 9500.426514] ? nd_jump_root+0xe5/0x160 [ 9500.436354] ? path_init+0x437/0x520 [ 9500.440059] path_openat+0xf7/0x500 [ 9500.449797] ? do_raw_spin_unlock+0x75/0x190 [ 9500.452066] do_filp_open+0x99/0x140 [ 9500.465846] ? getname_flags+0x6e/0x330 [ 9500.467072] ? __check_object_size+0xff/0x256 [ 9500.468607] ? do_raw_spin_unlock+0x75/0x190 [ 9500.484529] ? _raw_spin_unlock+0x12/0x30 [ 9500.500316] do_sys_openat2+0x2b4/0x410 [ 9500.501804] do_sys_open+0x73/0xa0 [ 9500.503129] __x64_sys_openat+0x24/0x30 [ 9500.522152] do_syscall_64+0xc1/0x440 [ 9500.527270] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9500.536303] RIP: 0033:0x7f1a63983332 [ 9500.544715] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9500.557448] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9500.564434] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9500.571475] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9500.577316] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9500.581971] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9500.587926] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9505.762632] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7482s! [ 9505.765364] Showing busy workqueues and worker pools: [ 9505.768711] workqueue events: flags=0x0 [ 9505.771083] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9505.776097] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9505.778575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9505.782317] pending: sec_gc_main [ptlrpc] [ 9505.784625] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9505.791202] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9505.797571] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9505.800053] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9505.804386] workqueue events_long: flags=0x0 [ 9505.806548] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9505.811588] pending: lu_objects_flush [obdclass] [ 9505.813859] workqueue events_unbound: flags=0x2 [ 9505.816951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9505.822001] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9505.829409] workqueue events_power_efficient: flags=0x80 [ 9505.833047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9505.836805] pending: neigh_periodic_work, do_cache_clean [ 9505.840296] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9505.843860] pending: neigh_periodic_work, check_lifetime [ 9505.846868] workqueue mm_percpu_wq: flags=0x8 [ 9505.850099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9505.854171] pending: vmstat_update [ 9505.855568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9505.861831] pending: vmstat_update [ 9505.863285] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9505.874342] pending: vmstat_update [ 9505.877393] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9505.885068] pending: vmstat_update [ 9505.887150] workqueue writeback: flags=0x4a [ 9505.889454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9505.898881] pending: wb_workfn, wb_workfn [ 9505.900401] workqueue xprtiod: flags=0xa [ 9505.901530] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9505.912279] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9505.918656] workqueue ipv6_addrconf: flags=0x40008 [ 9505.920414] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9505.936305] pending: addrconf_verify_work [ 9505.940709] workqueue knbd0-recv: flags=0x1a [ 9505.941893] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9505.947895] in-flight: 647:recv_work [ 9505.956171] workqueue ptlrpc_pinger: flags=0x2 [ 9505.957604] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9505.959448] pending: ptlrpc_pinger_main [ptlrpc] [ 9505.984342] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9496s workers=2 idle: 652 [ 9508.056028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 9508.057119] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 9508.059688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9508.059779] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9508.059786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9508.059789] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9508.059802] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9508.059806] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9508.059815] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9508.059820] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9508.059822] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9508.059824] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9508.059826] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9508.059829] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9508.059832] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9508.059835] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9508.066855] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9508.066944] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9508.066952] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9508.066955] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9508.066970] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9508.066974] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9508.066983] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9508.066985] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9508.066988] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9508.066990] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9508.066992] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9508.066996] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9508.066999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9508.067001] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9508.067010] Call Trace: [ 9508.067016] [ 9508.067021] ? show_regs.cold.9+0x22/0x2f [ 9508.067040] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9508.067047] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9508.067056] ? watchdog+0x40/0x40 [ 9508.067060] ? hrtimer_interrupt+0x128/0x2c0 [ 9508.067064] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9508.067071] ? apic_timer_interrupt+0xf/0x20 [ 9508.067076] [ 9508.067078] ? native_safe_halt+0x2e/0x50 [ 9508.067083] kvm_wait+0xcd/0xe0 [ 9508.067091] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9508.067099] do_raw_spin_lock+0x177/0x190 [ 9508.067104] _raw_spin_lock+0x21/0x30 [ 9508.067108] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9508.067449] ? do_raw_spin_unlock+0x75/0x190 [ 9508.067455] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9508.067673] ? do_wait_intr+0xf0/0xf0 [ 9508.067683] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9508.067927] mdc_close+0x53b/0xd70 [mdc] [ 9508.067973] lmv_close+0x18d/0x490 [lmv] [ 9508.068008] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9508.068153] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9508.068240] ll_md_close+0x274/0x800 [lustre] [ 9508.068336] ll_file_release+0x128/0xc50 [lustre] [ 9508.068426] __fput+0x10d/0x440 [ 9508.068435] ____fput+0x12/0x20 [ 9508.068440] task_work_run+0xc6/0x110 [ 9508.068452] exit_to_usermode_loop+0x1dd/0x1f0 [ 9508.068459] do_syscall_64+0x3d6/0x440 [ 9508.068467] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9508.068474] RIP: 0033:0x7f1a63983cd8 [ 9508.068481] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9508.068487] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9508.068492] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9508.068494] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9508.068497] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9508.068499] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9508.068501] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9508.358426] Call Trace: [ 9508.359684] [ 9508.360776] ? show_regs.cold.9+0x22/0x2f [ 9508.363052] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9508.365480] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9508.367674] ? watchdog+0x40/0x40 [ 9508.369405] ? hrtimer_interrupt+0x128/0x2c0 [ 9508.371569] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9508.374076] ? apic_timer_interrupt+0xf/0x20 [ 9508.376585] [ 9508.377576] ? native_safe_halt+0x2e/0x50 [ 9508.379595] kvm_wait+0xcd/0xe0 [ 9508.381513] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9508.384150] do_raw_spin_lock+0x177/0x190 [ 9508.386286] _raw_spin_lock+0x21/0x30 [ 9508.387789] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9508.390670] ? do_raw_spin_unlock+0x75/0x190 [ 9508.392878] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9508.395601] ? do_wait_intr+0xf0/0xf0 [ 9508.398820] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9508.402352] mdc_close+0x53b/0xd70 [mdc] [ 9508.404882] lmv_close+0x18d/0x490 [lmv] [ 9508.407485] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9508.411886] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9508.415396] ll_md_close+0x274/0x800 [lustre] [ 9508.418799] ll_file_release+0x128/0xc50 [lustre] [ 9508.421459] __fput+0x10d/0x440 [ 9508.422981] ____fput+0x12/0x20 [ 9508.424412] task_work_run+0xc6/0x110 [ 9508.426533] exit_to_usermode_loop+0x1dd/0x1f0 [ 9508.428572] do_syscall_64+0x3d6/0x440 [ 9508.430577] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9508.432963] RIP: 0033:0x7f1a63983cd8 [ 9508.434676] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9508.443067] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9508.446654] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9508.450234] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9508.453403] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9508.456558] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9508.459668] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9516.058062] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9516.061829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9516.082198] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9516.086951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9516.091071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9516.094299] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9516.109448] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9516.114966] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9516.122405] RDX: 0000000000003325 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9516.129687] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9516.134961] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9516.140360] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9516.146725] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9516.152445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9516.155960] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9516.158405] Call Trace: [ 9516.161098] [ 9516.162278] ? show_regs.cold.9+0x22/0x2f [ 9516.165374] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9516.168823] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9516.173490] ? watchdog+0x40/0x40 [ 9516.175401] ? hrtimer_interrupt+0x128/0x2c0 [ 9516.178890] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9516.180667] ? apic_timer_interrupt+0xf/0x20 [ 9516.184545] [ 9516.185582] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9516.191309] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9516.195331] do_raw_spin_lock+0x177/0x190 [ 9516.197948] _raw_spin_lock+0x21/0x30 [ 9516.200111] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9516.203742] obd_mod_free+0x46/0x140 [ptlrpc] [ 9516.206846] mdc_commit_open+0x91/0xb0 [mdc] [ 9516.212723] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9516.216320] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9516.220329] after_reply+0xe2c/0x1780 [ptlrpc] [ 9516.222532] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9516.225594] ? __next_timer_interrupt+0x160/0x160 [ 9516.227532] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9516.229250] ? do_wait_intr+0xf0/0xf0 [ 9516.232503] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9516.236383] kthread+0x1d1/0x200 [ 9516.238240] ? set_kthread_struct+0x70/0x70 [ 9516.240988] ret_from_fork+0x1f/0x30 [ 9532.053087] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9532.058942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9532.108819] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9532.115664] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9532.121034] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9532.123878] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9532.140518] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9532.145189] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9532.154173] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9532.167862] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9532.175269] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9532.180400] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9532.184445] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9532.190151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9532.195091] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9532.202853] Call Trace: [ 9532.204551] [ 9532.205403] ? show_regs.cold.9+0x22/0x2f [ 9532.209370] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9532.212280] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9532.216895] ? watchdog+0x40/0x40 [ 9532.223110] ? hrtimer_interrupt+0x128/0x2c0 [ 9532.227038] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9532.235521] ? apic_timer_interrupt+0xf/0x20 [ 9532.240303] [ 9532.242759] ? native_safe_halt+0x2e/0x50 [ 9532.248179] kvm_wait+0xcd/0xe0 [ 9532.252207] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9532.255696] do_raw_spin_lock+0x177/0x190 [ 9532.258128] _raw_spin_lock+0x21/0x30 [ 9532.262943] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9532.270837] ? do_raw_spin_unlock+0x75/0x190 [ 9532.273341] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9532.281416] ? do_wait_intr+0xf0/0xf0 [ 9532.292533] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9532.298664] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9532.304146] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9532.308504] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9532.312245] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9532.317940] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9532.330607] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9532.333000] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9532.342995] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9532.358286] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9532.367895] lmv_intent_open+0x338/0x1150 [lmv] [ 9532.375329] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9532.384255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9532.394932] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9532.401591] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9532.404517] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9532.417333] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9532.420380] ? do_raw_spin_unlock+0x75/0x190 [ 9532.422930] ? _raw_spin_unlock+0x12/0x30 [ 9532.425199] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9532.438679] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9532.442985] ll_atomic_open+0x36a/0xd70 [lustre] [ 9532.452924] lookup_open+0x356/0xb40 [ 9532.456344] do_last+0x313/0xfc0 [ 9532.459887] ? nd_jump_root+0xe5/0x160 [ 9532.461816] ? path_init+0x437/0x520 [ 9532.468550] path_openat+0xf7/0x500 [ 9532.470291] ? do_raw_spin_unlock+0x75/0x190 [ 9532.476551] do_filp_open+0x99/0x140 [ 9532.482179] ? getname_flags+0x6e/0x330 [ 9532.485402] ? __check_object_size+0xff/0x256 [ 9532.489087] ? do_raw_spin_unlock+0x75/0x190 [ 9532.494652] ? _raw_spin_unlock+0x12/0x30 [ 9532.497906] do_sys_openat2+0x2b4/0x410 [ 9532.499918] do_sys_open+0x73/0xa0 [ 9532.502304] __x64_sys_openat+0x24/0x30 [ 9532.504604] do_syscall_64+0xc1/0x440 [ 9532.506806] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9532.508798] RIP: 0033:0x7f1a63983332 [ 9532.510609] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9532.519928] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9532.523907] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9532.528718] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9532.538220] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9532.542723] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9532.550136] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9536.058189] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 9536.058689] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 9536.061275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9536.061353] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9536.061361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9536.061364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.061378] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9536.061382] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9536.061387] RAX: 0000000000004728 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9536.061389] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9536.061392] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9536.061394] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9536.061397] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9536.061400] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9536.061403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9536.061405] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9536.061413] Call Trace: [ 9536.061418] [ 9536.061423] ? show_regs.cold.9+0x22/0x2f [ 9536.061445] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9536.061452] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9536.061456] ? watchdog+0x40/0x40 [ 9536.061460] ? hrtimer_interrupt+0x128/0x2c0 [ 9536.061465] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9536.061473] ? apic_timer_interrupt+0xf/0x20 [ 9536.061477] [ 9536.061481] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.061487] do_raw_spin_lock+0x177/0x190 [ 9536.061492] _raw_spin_lock+0x21/0x30 [ 9536.061497] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9536.061757] ? do_raw_spin_unlock+0x75/0x190 [ 9536.061762] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9536.061961] ? do_wait_intr+0xf0/0xf0 [ 9536.061969] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9536.062163] mdc_close+0x53b/0xd70 [mdc] [ 9536.062202] lmv_close+0x18d/0x490 [lmv] [ 9536.062233] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9536.062340] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9536.062422] ll_md_close+0x274/0x800 [lustre] [ 9536.062517] ll_file_release+0x128/0xc50 [lustre] [ 9536.062597] __fput+0x10d/0x440 [ 9536.062606] ____fput+0x12/0x20 [ 9536.062610] task_work_run+0xc6/0x110 [ 9536.062617] exit_to_usermode_loop+0x1dd/0x1f0 [ 9536.062624] do_syscall_64+0x3d6/0x440 [ 9536.062629] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9536.062635] RIP: 0033:0x7f1a63983cd8 [ 9536.062642] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9536.062646] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9536.062651] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9536.062653] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9536.062655] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9536.062657] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9536.062659] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9536.305642] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9536.320868] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9536.324487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9536.327383] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.329213] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9536.335168] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9536.343790] RAX: 0000000000005c10 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9536.347649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9536.350199] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9536.352985] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9536.355144] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9536.357538] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9536.359915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9536.362224] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9536.365970] Call Trace: [ 9536.366713] [ 9536.367255] ? show_regs.cold.9+0x22/0x2f [ 9536.368375] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9536.372319] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9536.374269] ? watchdog+0x40/0x40 [ 9536.375499] ? hrtimer_interrupt+0x128/0x2c0 [ 9536.376739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9536.380285] ? apic_timer_interrupt+0xf/0x20 [ 9536.381547] [ 9536.382056] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.388088] do_raw_spin_lock+0x177/0x190 [ 9536.389683] _raw_spin_lock+0x21/0x30 [ 9536.391627] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9536.393249] ? do_raw_spin_unlock+0x75/0x190 [ 9536.399068] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9536.401497] ? do_wait_intr+0xf0/0xf0 [ 9536.402939] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9536.404849] mdc_close+0x53b/0xd70 [mdc] [ 9536.406014] lmv_close+0x18d/0x490 [lmv] [ 9536.408628] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9536.413748] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9536.416231] ll_md_close+0x274/0x800 [lustre] [ 9536.417656] ll_file_release+0x128/0xc50 [lustre] [ 9536.419083] __fput+0x10d/0x440 [ 9536.420190] ____fput+0x12/0x20 [ 9536.421191] task_work_run+0xc6/0x110 [ 9536.422290] exit_to_usermode_loop+0x1dd/0x1f0 [ 9536.427578] do_syscall_64+0x3d6/0x440 [ 9536.429186] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9536.431015] RIP: 0033:0x7f1a63983cd8 [ 9536.432089] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9536.441914] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9536.448818] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9536.454497] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9536.463191] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9536.470204] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9536.475167] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9536.488278] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7513s! [ 9536.496270] Showing busy workqueues and worker pools: [ 9536.500030] workqueue events: flags=0x0 [ 9536.504493] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9536.508805] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9536.514342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9536.517789] pending: sec_gc_main [ptlrpc] [ 9536.520533] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9536.525074] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9536.534107] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9536.544970] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9536.555828] workqueue events_long: flags=0x0 [ 9536.559449] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9536.564046] pending: lu_objects_flush [obdclass] [ 9536.568894] workqueue events_unbound: flags=0x2 [ 9536.572313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9536.587893] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9536.597876] workqueue events_power_efficient: flags=0x80 [ 9536.605603] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9536.615875] pending: neigh_periodic_work, do_cache_clean [ 9536.617983] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9536.637321] pending: neigh_periodic_work, check_lifetime [ 9536.643896] workqueue mm_percpu_wq: flags=0x8 [ 9536.653800] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9536.661577] pending: vmstat_update [ 9536.668168] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9536.670614] pending: vmstat_update [ 9536.678923] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9536.687845] pending: vmstat_update [ 9536.690993] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9536.696129] pending: vmstat_update [ 9536.697385] workqueue writeback: flags=0x4a [ 9536.699724] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9536.707761] pending: wb_workfn, wb_workfn [ 9536.714981] workqueue xprtiod: flags=0xa [ 9536.717406] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9536.730394] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9536.743531] workqueue ipv6_addrconf: flags=0x40008 [ 9536.749096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9536.753985] pending: addrconf_verify_work [ 9536.756685] workqueue knbd0-recv: flags=0x1a [ 9536.763544] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9536.773986] in-flight: 647:recv_work [ 9536.779815] workqueue ptlrpc_pinger: flags=0x2 [ 9536.789862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9536.807756] pending: ptlrpc_pinger_main [ptlrpc] [ 9536.819404] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9526s workers=2 idle: 652 [ 9544.058060] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9544.064361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9544.097085] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9544.111423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9544.118130] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9544.123098] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9544.137413] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9544.140954] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9544.144573] RDX: 0000000000006247 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9544.147318] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9544.150294] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9544.152976] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9544.155201] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9544.160967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9544.163756] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9544.166529] Call Trace: [ 9544.167235] [ 9544.168249] ? show_regs.cold.9+0x22/0x2f [ 9544.169309] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9544.171274] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9544.174144] ? watchdog+0x40/0x40 [ 9544.175033] ? hrtimer_interrupt+0x128/0x2c0 [ 9544.176151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9544.178757] ? apic_timer_interrupt+0xf/0x20 [ 9544.179828] [ 9544.180351] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9544.182517] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9544.183931] do_raw_spin_lock+0x177/0x190 [ 9544.184967] _raw_spin_lock+0x21/0x30 [ 9544.186132] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9544.187794] obd_mod_free+0x46/0x140 [ptlrpc] [ 9544.189192] mdc_commit_open+0x91/0xb0 [mdc] [ 9544.191254] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9544.196667] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9544.199082] after_reply+0xe2c/0x1780 [ptlrpc] [ 9544.201345] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9544.203927] ? __next_timer_interrupt+0x160/0x160 [ 9544.206655] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9544.208434] ? do_wait_intr+0xf0/0xf0 [ 9544.209806] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9544.211885] kthread+0x1d1/0x200 [ 9544.213153] ? set_kthread_struct+0x70/0x70 [ 9544.214781] ret_from_fork+0x1f/0x30 [ 9564.053133] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 9564.057034] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 9564.057106] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 9564.057113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9564.057185] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9564.057190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9564.057193] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9564.057207] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9564.057212] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9564.057216] RAX: 0000000000006987 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9564.057218] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9564.057221] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9564.057223] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9564.057224] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9564.057228] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9564.057230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9564.057232] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9564.057322] Call Trace: [ 9564.057326] [ 9564.057331] ? show_regs.cold.9+0x22/0x2f [ 9564.057341] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9564.057348] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9564.057351] ? watchdog+0x40/0x40 [ 9564.057355] ? hrtimer_interrupt+0x128/0x2c0 [ 9564.057359] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9564.057366] ? apic_timer_interrupt+0xf/0x20 [ 9564.057385] [ 9564.057387] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9564.057392] do_raw_spin_lock+0x177/0x190 [ 9564.057396] _raw_spin_lock+0x21/0x30 [ 9564.057400] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9564.057658] ? do_raw_spin_unlock+0x75/0x190 [ 9564.057664] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9564.057866] ? do_wait_intr+0xf0/0xf0 [ 9564.057881] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9564.058233] mdc_close+0x53b/0xd70 [mdc] [ 9564.058285] lmv_close+0x18d/0x490 [lmv] [ 9564.058318] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9564.058426] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9564.059494] ll_md_close+0x274/0x800 [lustre] [ 9564.060017] ll_file_release+0x128/0xc50 [lustre] [ 9564.060599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 9564.060609] __fput+0x10d/0x440 [ 9564.060616] mdd(O) mgs(O) lquota(O) lfsck(O) [ 9564.060624] ____fput+0x12/0x20 [ 9564.060628] obdecho(O) mgc(O) mdc(O) lov(O) [ 9564.060634] task_work_run+0xc6/0x110 [ 9564.060640] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 9564.060647] exit_to_usermode_loop+0x1dd/0x1f0 [ 9564.060654] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [ 9564.060661] do_syscall_64+0x3d6/0x440 [ 9564.060667] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 9564.060674] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9564.060682] intel_rapl_common sb_edac rapl pcspkr [ 9564.060689] RIP: 0033:0x7f1a63983cd8 [ 9564.060693] i2c_piix4 squashfs crct10dif_pclmul [ 9564.060702] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9564.060708] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9564.060714] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9564.060716] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9564.060718] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9564.060720] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9564.060723] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9564.060701] crc32_pclmul [ 9564.066176] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9564.066258] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9564.066266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9564.066269] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9564.066288] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9564.066293] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9564.066297] RAX: 0000000000006605 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9564.066300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9564.066302] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9564.066304] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9564.066306] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9564.066309] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9564.066312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9564.066314] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9564.066321] Call Trace: [ 9564.066326] [ 9564.066332] ? show_regs.cold.9+0x22/0x2f [ 9564.066827] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9564.066834] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9564.066839] ? watchdog+0x40/0x40 [ 9564.066844] ? hrtimer_interrupt+0x128/0x2c0 [ 9564.066848] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9564.066856] ? apic_timer_interrupt+0xf/0x20 [ 9564.066860] [ 9564.066863] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9564.066869] do_raw_spin_lock+0x177/0x190 [ 9564.066873] _raw_spin_lock+0x21/0x30 [ 9564.066883] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9564.067202] ? do_raw_spin_unlock+0x75/0x190 [ 9564.067208] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9564.075440] crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [ 9564.098869] ? do_wait_intr+0xf0/0xf0 [ 9564.098893] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9564.115559] sha512_ssse3 [ 9564.121082] mdc_close+0x53b/0xd70 [mdc] [ 9564.121140] lmv_close+0x18d/0x490 [lmv] [ 9564.121235] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9564.121471] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9564.121607] ll_md_close+0x274/0x800 [lustre] [ 9564.121693] ll_file_release+0x128/0xc50 [lustre] [ 9564.121777] __fput+0x10d/0x440 [ 9564.121788] ____fput+0x12/0x20 [ 9564.121793] task_work_run+0xc6/0x110 [ 9564.121801] exit_to_usermode_loop+0x1dd/0x1f0 [ 9564.121807] do_syscall_64+0x3d6/0x440 [ 9564.121812] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9564.121821] RIP: 0033:0x7f1a63983cd8 [ 9564.121828] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9564.121832] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9564.121838] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9564.121840] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9564.121842] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9564.121845] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9564.121847] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9564.453532] sha512_generic [ 9564.454840] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9564.463115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9564.469557] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9564.472764] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9564.490881] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9564.496889] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9564.501855] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9564.506130] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9564.509901] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9564.516626] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9564.521023] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9564.529850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9564.533864] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9564.537975] Call Trace: [ 9564.539121] [ 9564.540004] ? show_regs.cold.9+0x22/0x2f [ 9564.542002] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9564.544696] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9564.547086] ? watchdog+0x40/0x40 [ 9564.548895] ? hrtimer_interrupt+0x128/0x2c0 [ 9564.552273] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9564.554779] ? apic_timer_interrupt+0xf/0x20 [ 9564.557291] [ 9564.558462] ? native_safe_halt+0x2e/0x50 [ 9564.560934] kvm_wait+0xcd/0xe0 [ 9564.562120] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9564.563973] do_raw_spin_lock+0x177/0x190 [ 9564.566082] _raw_spin_lock+0x21/0x30 [ 9564.568536] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9564.571795] ? do_raw_spin_unlock+0x75/0x190 [ 9564.575656] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9564.579063] ? do_wait_intr+0xf0/0xf0 [ 9564.580454] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9564.590470] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9564.593954] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9564.601550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9564.604889] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9564.607486] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9564.611910] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9564.613407] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9564.615648] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9564.618281] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9564.620683] lmv_intent_open+0x338/0x1150 [lmv] [ 9564.622656] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9564.624761] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9564.627403] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9564.629038] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9564.631223] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9564.633914] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9564.636455] ? do_raw_spin_unlock+0x75/0x190 [ 9564.638389] ? _raw_spin_unlock+0x12/0x30 [ 9564.640426] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9564.643030] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9564.647991] ll_atomic_open+0x36a/0xd70 [lustre] [ 9564.651705] lookup_open+0x356/0xb40 [ 9564.653755] do_last+0x313/0xfc0 [ 9564.655987] ? nd_jump_root+0xe5/0x160 [ 9564.658328] ? path_init+0x437/0x520 [ 9564.660533] path_openat+0xf7/0x500 [ 9564.662850] ? do_raw_spin_unlock+0x75/0x190 [ 9564.665267] do_filp_open+0x99/0x140 [ 9564.666740] ? getname_flags+0x6e/0x330 [ 9564.668987] ? __check_object_size+0xff/0x256 [ 9564.670970] ? do_raw_spin_unlock+0x75/0x190 [ 9564.672923] ? _raw_spin_unlock+0x12/0x30 [ 9564.675117] do_sys_openat2+0x2b4/0x410 [ 9564.677302] do_sys_open+0x73/0xa0 [ 9564.678701] __x64_sys_openat+0x24/0x30 [ 9564.680845] do_syscall_64+0xc1/0x440 [ 9564.682087] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9564.684686] RIP: 0033:0x7f1a63983332 [ 9564.686099] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9564.692577] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9564.695727] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9564.698563] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9564.703820] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9564.707541] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9564.710386] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9567.201142] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7544s! [ 9567.205782] Showing busy workqueues and worker pools: [ 9567.208192] workqueue events: flags=0x0 [ 9567.210065] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9567.213568] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9567.217567] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9567.220675] pending: sec_gc_main [ptlrpc] [ 9567.223164] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9567.226695] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9567.232716] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9567.235980] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9567.242583] workqueue events_long: flags=0x0 [ 9567.244547] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9567.247635] pending: lu_objects_flush [obdclass] [ 9567.250616] workqueue events_unbound: flags=0x2 [ 9567.253187] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9567.255949] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9567.261274] workqueue events_power_efficient: flags=0x80 [ 9567.264720] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9567.266803] pending: neigh_periodic_work, do_cache_clean [ 9567.268987] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9567.271455] pending: neigh_periodic_work, check_lifetime [ 9567.274181] workqueue mm_percpu_wq: flags=0x8 [ 9567.276572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9567.279532] pending: vmstat_update [ 9567.280852] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9567.283286] pending: vmstat_update [ 9567.284714] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9567.288778] pending: vmstat_update [ 9567.290493] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9567.292699] pending: vmstat_update [ 9567.293904] workqueue writeback: flags=0x4a [ 9567.295458] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9567.299921] pending: wb_workfn, wb_workfn [ 9567.303725] workqueue xprtiod: flags=0xa [ 9567.306757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9567.311517] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9567.314676] workqueue ipv6_addrconf: flags=0x40008 [ 9567.316655] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9567.318676] pending: addrconf_verify_work [ 9567.320078] workqueue knbd0-recv: flags=0x1a [ 9567.322620] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9567.324625] in-flight: 647:recv_work [ 9567.325921] workqueue ptlrpc_pinger: flags=0x2 [ 9567.327555] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9567.329568] pending: ptlrpc_pinger_main [ptlrpc] [ 9567.331411] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9557s workers=2 idle: 652 [ 9572.058052] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9572.060510] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9572.076154] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9572.079606] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9572.084420] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9572.088497] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9572.100404] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9572.104842] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9572.109207] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9572.113690] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9572.118122] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9572.123276] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9572.127523] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9572.132168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9572.142524] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9572.150359] Call Trace: [ 9572.151732] [ 9572.152686] ? show_regs.cold.9+0x22/0x2f [ 9572.157983] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9572.164336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9572.166364] ? watchdog+0x40/0x40 [ 9572.169696] ? hrtimer_interrupt+0x128/0x2c0 [ 9572.172987] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9572.175038] ? apic_timer_interrupt+0xf/0x20 [ 9572.177915] [ 9572.178788] ? native_safe_halt+0x2e/0x50 [ 9572.182478] kvm_wait+0xcd/0xe0 [ 9572.184368] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9572.188473] do_raw_spin_lock+0x177/0x190 [ 9572.196943] _raw_spin_lock+0x21/0x30 [ 9572.199687] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9572.204556] obd_mod_free+0x46/0x140 [ptlrpc] [ 9572.207960] mdc_commit_open+0x91/0xb0 [mdc] [ 9572.213132] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9572.218245] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9572.220201] after_reply+0xe2c/0x1780 [ptlrpc] [ 9572.223931] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9572.232750] ? __next_timer_interrupt+0x160/0x160 [ 9572.235728] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9572.240310] ? do_wait_intr+0xf0/0xf0 [ 9572.243396] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9572.249797] kthread+0x1d1/0x200 [ 9572.254097] ? set_kthread_struct+0x70/0x70 [ 9572.255547] ret_from_fork+0x1f/0x30 [ 9592.058161] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 9592.061528] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 9592.062935] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 9592.062947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9592.063598] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9592.063603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9592.063607] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9592.063631] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9592.063635] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9592.063640] RAX: 0000000000000e86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9592.063643] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9592.063645] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9592.063647] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9592.063649] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9592.063652] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9592.063655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9592.063658] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9592.063668] Call Trace: [ 9592.063681] [ 9592.063688] ? show_regs.cold.9+0x22/0x2f [ 9592.063698] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9592.063706] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9592.063711] ? watchdog+0x40/0x40 [ 9592.063715] ? hrtimer_interrupt+0x128/0x2c0 [ 9592.063718] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9592.063727] ? apic_timer_interrupt+0xf/0x20 [ 9592.063731] [ 9592.063734] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9592.063739] do_raw_spin_lock+0x177/0x190 [ 9592.063744] _raw_spin_lock+0x21/0x30 [ 9592.063749] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9592.064285] ? do_raw_spin_unlock+0x75/0x190 [ 9592.064292] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9592.064961] ? do_wait_intr+0xf0/0xf0 [ 9592.065054] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9592.065643] mdc_close+0x53b/0xd70 [mdc] [ 9592.065682] lmv_close+0x18d/0x490 [lmv] [ 9592.065714] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9592.065828] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9592.065929] ll_md_close+0x274/0x800 [lustre] [ 9592.066031] ll_file_release+0x128/0xc50 [lustre] [ 9592.066275] __fput+0x10d/0x440 [ 9592.066284] ____fput+0x12/0x20 [ 9592.066289] task_work_run+0xc6/0x110 [ 9592.066304] exit_to_usermode_loop+0x1dd/0x1f0 [ 9592.066313] do_syscall_64+0x3d6/0x440 [ 9592.066317] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9592.066325] RIP: 0033:0x7f1a63983cd8 [ 9592.066348] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9592.066351] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9592.066356] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9592.066359] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9592.066361] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9592.066363] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9592.066365] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9592.068556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9592.068634] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9592.068642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9592.068646] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9592.068659] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9592.068663] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9592.068668] RAX: 0000000000003b09 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9592.068670] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9592.068673] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9592.068674] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9592.068677] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9592.068680] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9592.068683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9592.068685] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9592.068694] Call Trace: [ 9592.068699] [ 9592.068704] ? show_regs.cold.9+0x22/0x2f [ 9592.068714] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9592.068720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9592.068724] ? watchdog+0x40/0x40 [ 9592.068728] ? hrtimer_interrupt+0x128/0x2c0 [ 9592.068731] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9592.068740] ? apic_timer_interrupt+0xf/0x20 [ 9592.068744] [ 9592.068747] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9592.068752] do_raw_spin_lock+0x177/0x190 [ 9592.068757] _raw_spin_lock+0x21/0x30 [ 9592.068762] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9592.095267] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9592.095346] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9592.095354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9592.095357] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9592.095370] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9592.095375] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9592.095379] RAX: 0000000000007314 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9592.095381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9592.095384] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9592.095386] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9592.095389] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9592.095392] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9592.095396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9592.095398] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9592.095407] Call Trace: [ 9592.095413] [ 9592.095419] ? show_regs.cold.9+0x22/0x2f [ 9592.095429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9592.095437] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9592.095441] ? watchdog+0x40/0x40 [ 9592.095445] ? hrtimer_interrupt+0x128/0x2c0 [ 9592.095449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9592.095459] ? apic_timer_interrupt+0xf/0x20 [ 9592.095464] [ 9592.095467] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9592.095474] do_raw_spin_lock+0x177/0x190 [ 9592.095479] _raw_spin_lock+0x21/0x30 [ 9592.095484] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9592.096532] ? do_raw_spin_unlock+0x75/0x190 [ 9592.096539] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9592.097273] ? do_wait_intr+0xf0/0xf0 [ 9592.097282] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9592.097492] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9592.098058] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9592.098272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9592.116741] ? do_raw_spin_unlock+0x75/0x190 [ 9592.116770] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9592.118024] ? do_wait_intr+0xf0/0xf0 [ 9592.118034] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9592.118229] mdc_close+0x53b/0xd70 [mdc] [ 9592.205625] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9592.205999] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9592.206034] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9592.206065] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9592.206193] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9592.206498] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9592.206530] lmv_intent_open+0x338/0x1150 [lmv] [ 9592.206565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9592.206659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9592.206966] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9592.206999] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9592.207080] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9592.207170] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9592.207249] ? do_raw_spin_unlock+0x75/0x190 [ 9592.207256] ? _raw_spin_unlock+0x12/0x30 [ 9592.207262] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9592.207290] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9592.207309] ll_atomic_open+0x36a/0xd70 [lustre] [ 9592.207391] lookup_open+0x356/0xb40 [ 9592.207401] do_last+0x313/0xfc0 [ 9592.207406] ? nd_jump_root+0xe5/0x160 [ 9592.207410] ? path_init+0x437/0x520 [ 9592.207414] path_openat+0xf7/0x500 [ 9592.207419] ? do_raw_spin_unlock+0x75/0x190 [ 9592.207425] do_filp_open+0x99/0x140 [ 9592.207431] ? getname_flags+0x6e/0x330 [ 9592.207436] ? __check_object_size+0xff/0x256 [ 9592.207443] ? do_raw_spin_unlock+0x75/0x190 [ 9592.207447] ? _raw_spin_unlock+0x12/0x30 [ 9592.207452] do_sys_openat2+0x2b4/0x410 [ 9592.207456] do_sys_open+0x73/0xa0 [ 9592.207461] __x64_sys_openat+0x24/0x30 [ 9592.207465] do_syscall_64+0xc1/0x440 [ 9592.207473] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9592.207480] RIP: 0033:0x7f1a63983332 [ 9592.207487] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9592.207490] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9592.207495] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9592.207498] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9592.207500] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9592.207502] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9592.207504] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9593.393428] lmv_close+0x18d/0x490 [lmv] [ 9593.394809] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9593.397182] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9593.398963] ll_md_close+0x274/0x800 [lustre] [ 9593.407032] ll_file_release+0x128/0xc50 [lustre] [ 9593.408835] __fput+0x10d/0x440 [ 9593.419743] ____fput+0x12/0x20 [ 9593.420747] task_work_run+0xc6/0x110 [ 9593.421739] exit_to_usermode_loop+0x1dd/0x1f0 [ 9593.430524] do_syscall_64+0x3d6/0x440 [ 9593.436822] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9593.439845] RIP: 0033:0x7f1a63983cd8 [ 9593.443605] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9593.466150] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9593.474164] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9593.477807] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9593.483450] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9593.488082] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9593.491978] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9597.419150] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7574s! [ 9597.426752] Showing busy workqueues and worker pools: [ 9597.429699] workqueue events: flags=0x0 [ 9597.433673] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9597.437640] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9597.444987] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9597.454630] pending: sec_gc_main [ptlrpc] [ 9597.463531] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9597.468545] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9597.482177] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9597.490611] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9597.498388] workqueue events_long: flags=0x0 [ 9597.502205] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9597.506523] pending: lu_objects_flush [obdclass] [ 9597.510715] workqueue events_unbound: flags=0x2 [ 9597.513345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9597.517322] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9597.523663] workqueue events_power_efficient: flags=0x80 [ 9597.527108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9597.530272] pending: neigh_periodic_work, do_cache_clean [ 9597.536236] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9597.544369] pending: neigh_periodic_work, check_lifetime [ 9597.548099] workqueue mm_percpu_wq: flags=0x8 [ 9597.550503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9597.560985] pending: vmstat_update [ 9597.564169] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9597.571758] pending: vmstat_update [ 9597.573581] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9597.576519] pending: vmstat_update [ 9597.582174] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9597.587081] pending: vmstat_update [ 9597.588345] workqueue writeback: flags=0x4a [ 9597.596317] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9597.605206] pending: wb_workfn, wb_workfn [ 9597.612123] workqueue xprtiod: flags=0xa [ 9597.613529] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9597.617500] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9597.622516] workqueue ipv6_addrconf: flags=0x40008 [ 9597.624672] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9597.628585] pending: addrconf_verify_work [ 9597.633415] workqueue knbd0-recv: flags=0x1a [ 9597.635585] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9597.642334] in-flight: 647:recv_work [ 9597.648242] workqueue ptlrpc_pinger: flags=0x2 [ 9597.652663] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9597.660550] pending: ptlrpc_pinger_main [ptlrpc] [ 9597.670582] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9587s workers=2 idle: 652 [ 9600.059760] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9600.068898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9600.146456] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9600.164502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9600.174288] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9600.178759] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9600.192227] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9600.195333] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9600.198849] RDX: 0000000000001f2d RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9600.202314] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9600.205663] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9600.209173] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9600.212616] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9600.216587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9600.219322] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9600.222846] Call Trace: [ 9600.224572] [ 9600.225747] ? show_regs.cold.9+0x22/0x2f [ 9600.227596] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9600.230176] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9600.232665] ? watchdog+0x40/0x40 [ 9600.234432] ? hrtimer_interrupt+0x128/0x2c0 [ 9600.238244] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9600.240739] ? apic_timer_interrupt+0xf/0x20 [ 9600.242705] [ 9600.243709] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9600.246356] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9600.248695] do_raw_spin_lock+0x177/0x190 [ 9600.250711] _raw_spin_lock+0x21/0x30 [ 9600.252531] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9600.255583] obd_mod_free+0x46/0x140 [ptlrpc] [ 9600.258456] mdc_commit_open+0x91/0xb0 [mdc] [ 9600.260498] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9600.262798] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9600.265543] after_reply+0xe2c/0x1780 [ptlrpc] [ 9600.268172] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9600.295080] ? __next_timer_interrupt+0x160/0x160 [ 9600.305139] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9600.308193] ? do_wait_intr+0xf0/0xf0 [ 9600.311628] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9600.324770] kthread+0x1d1/0x200 [ 9600.328355] ? set_kthread_struct+0x70/0x70 [ 9600.334922] ret_from_fork+0x1f/0x30 [ 9620.056041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9620.057020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9620.059044] Modules linked in: [ 9620.065813] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9620.066429] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9620.066436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9620.066439] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9620.066452] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9620.066456] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9620.066461] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9620.066463] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9620.066465] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9620.066467] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9620.066469] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9620.066474] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9620.071434] lustre(O) [ 9620.094708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9620.094723] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9620.094732] Call Trace: [ 9620.094738] [ 9620.094744] ? show_regs.cold.9+0x22/0x2f [ 9620.094762] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9620.094769] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9620.094779] ? watchdog+0x40/0x40 [ 9620.099259] osp(O) [ 9620.107487] ? hrtimer_interrupt+0x128/0x2c0 [ 9620.107506] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9620.107530] ? apic_timer_interrupt+0xf/0x20 [ 9620.107538] [ 9620.107543] ? native_safe_halt+0x2e/0x50 [ 9620.107547] kvm_wait+0xcd/0xe0 [ 9620.107559] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9620.107568] do_raw_spin_lock+0x177/0x190 [ 9620.107573] _raw_spin_lock+0x21/0x30 [ 9620.107578] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9620.108142] ? do_raw_spin_unlock+0x75/0x190 [ 9620.108148] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9620.108436] ? do_wait_intr+0xf0/0xf0 [ 9620.108446] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9620.108830] mdc_close+0x53b/0xd70 [mdc] [ 9620.108976] lmv_close+0x18d/0x490 [lmv] [ 9620.109015] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9620.109153] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9620.109246] ll_md_close+0x274/0x800 [lustre] [ 9620.109342] ll_file_release+0x128/0xc50 [lustre] [ 9620.109437] __fput+0x10d/0x440 [ 9620.109447] ____fput+0x12/0x20 [ 9620.109452] task_work_run+0xc6/0x110 [ 9620.109460] exit_to_usermode_loop+0x1dd/0x1f0 [ 9620.109466] do_syscall_64+0x3d6/0x440 [ 9620.109471] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9620.109478] RIP: 0033:0x7f1a63983cd8 [ 9620.109485] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9620.109490] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9620.109494] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9620.109497] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9620.109499] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9620.109501] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9620.109503] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9620.236229] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9620.250322] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9620.255218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9620.258434] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9620.260430] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9620.267060] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9620.269801] RAX: 000000000000100a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9620.272193] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9620.274850] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9620.276997] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9620.279265] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9620.281550] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9620.284354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9620.286304] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9620.288756] Call Trace: [ 9620.289535] [ 9620.290216] ? show_regs.cold.9+0x22/0x2f [ 9620.291383] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9620.293013] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9620.294328] ? watchdog+0x40/0x40 [ 9620.295394] ? hrtimer_interrupt+0x128/0x2c0 [ 9620.296705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9620.298346] ? apic_timer_interrupt+0xf/0x20 [ 9620.299732] [ 9620.300404] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9620.302408] do_raw_spin_lock+0x177/0x190 [ 9620.303664] _raw_spin_lock+0x21/0x30 [ 9620.304813] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9620.306837] ? do_raw_spin_unlock+0x75/0x190 [ 9620.308225] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9620.310402] ? do_wait_intr+0xf0/0xf0 [ 9620.311770] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9620.313640] mdc_close+0x53b/0xd70 [mdc] [ 9620.315010] lmv_close+0x18d/0x490 [lmv] [ 9620.316190] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9620.318178] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9620.319902] ll_md_close+0x274/0x800 [lustre] [ 9620.321279] ll_file_release+0x128/0xc50 [lustre] [ 9620.322867] __fput+0x10d/0x440 [ 9620.324118] ____fput+0x12/0x20 [ 9620.325130] task_work_run+0xc6/0x110 [ 9620.326412] exit_to_usermode_loop+0x1dd/0x1f0 [ 9620.328137] do_syscall_64+0x3d6/0x440 [ 9620.329677] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9620.331410] RIP: 0033:0x7f1a63983cd8 [ 9620.332877] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9620.339662] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9620.342267] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9620.344933] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9620.347215] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9620.349856] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9620.352464] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9624.053030] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 9624.058855] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9624.097487] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9624.110882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9624.117008] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9624.123291] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9624.135422] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9624.154367] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9624.166163] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9624.176941] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9624.182147] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9624.186074] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9624.191953] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9624.196445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9624.202931] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9624.209141] Call Trace: [ 9624.217217] [ 9624.217950] ? show_regs.cold.9+0x22/0x2f [ 9624.219354] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9624.227433] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9624.232010] ? watchdog+0x40/0x40 [ 9624.233948] ? hrtimer_interrupt+0x128/0x2c0 [ 9624.239990] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9624.244294] ? apic_timer_interrupt+0xf/0x20 [ 9624.251073] [ 9624.252390] ? native_safe_halt+0x2e/0x50 [ 9624.256397] kvm_wait+0xcd/0xe0 [ 9624.262486] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9624.268038] do_raw_spin_lock+0x177/0x190 [ 9624.271048] _raw_spin_lock+0x21/0x30 [ 9624.273193] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9624.276415] ? do_raw_spin_unlock+0x75/0x190 [ 9624.278524] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9624.281172] ? do_wait_intr+0xf0/0xf0 [ 9624.283029] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9624.286758] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9624.290354] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9624.294686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9624.300943] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9624.306057] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9624.311273] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9624.318372] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9624.320302] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9624.328893] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9624.339181] lmv_intent_open+0x338/0x1150 [lmv] [ 9624.340943] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9624.346050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9624.349888] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9624.352095] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9624.356308] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9624.358391] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9624.362114] ? do_raw_spin_unlock+0x75/0x190 [ 9624.367518] ? _raw_spin_unlock+0x12/0x30 [ 9624.373365] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9624.377344] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9624.380162] ll_atomic_open+0x36a/0xd70 [lustre] [ 9624.389893] lookup_open+0x356/0xb40 [ 9624.391217] do_last+0x313/0xfc0 [ 9624.395928] ? nd_jump_root+0xe5/0x160 [ 9624.399479] ? path_init+0x437/0x520 [ 9624.408077] path_openat+0xf7/0x500 [ 9624.414669] ? do_raw_spin_unlock+0x75/0x190 [ 9624.419918] do_filp_open+0x99/0x140 [ 9624.422345] ? getname_flags+0x6e/0x330 [ 9624.432207] ? __check_object_size+0xff/0x256 [ 9624.433800] ? do_raw_spin_unlock+0x75/0x190 [ 9624.438164] ? _raw_spin_unlock+0x12/0x30 [ 9624.443106] do_sys_openat2+0x2b4/0x410 [ 9624.452901] do_sys_open+0x73/0xa0 [ 9624.454174] __x64_sys_openat+0x24/0x30 [ 9624.459628] do_syscall_64+0xc1/0x440 [ 9624.465367] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9624.475782] RIP: 0033:0x7f1a63983332 [ 9624.477082] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9624.506152] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9624.521926] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9624.524238] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9624.538451] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9624.557589] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9624.568153] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9628.058288] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9628.066257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9628.108670] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9628.121910] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9628.126850] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9628.128048] BUG: workqueue lockup - pool [ 9628.132136] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9628.132150] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 9628.134442] cpus=0-3 flags=0x4 nice=0 stuck for 7605s! [ 9628.134460] Showing busy workqueues and worker pools: [ 9628.134462] workqueue events: flags=0x0 [ 9628.134465] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9628.134475] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9628.134881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9628.134890] pending: sec_gc_main [ptlrpc] [ 9628.135198] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9628.135209] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9628.135526] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9628.135644] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9628.136131] workqueue events_long: flags=0x0 [ 9628.136135] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9628.136142] pending: lu_objects_flush [obdclass] [ 9628.136322] workqueue events_unbound: flags=0x2 [ 9628.136325] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9628.136332] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9628.136356] workqueue events_power_efficient: flags=0x80 [ 9628.136359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9628.136366] pending: neigh_periodic_work, do_cache_clean [ 9628.136379] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9628.136385] pending: neigh_periodic_work, check_lifetime [ 9628.136399] workqueue mm_percpu_wq: flags=0x8 [ 9628.136402] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9628.136407] pending: vmstat_update [ 9628.136412] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9628.136417] pending: vmstat_update [ 9628.136421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9628.136426] pending: vmstat_update [ 9628.136430] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9628.136435] pending: vmstat_update [ 9628.136443] workqueue writeback: flags=0x4a [ 9628.136445] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9628.136449] pending: wb_workfn, wb_workfn [ 9628.136462] workqueue xprtiod: flags=0xa [ 9628.136464] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9628.136469] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9628.136487] workqueue ipv6_addrconf: flags=0x40008 [ 9628.136489] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9628.136495] pending: addrconf_verify_work [ 9628.136506] workqueue knbd0-recv: flags=0x1a [ 9628.136508] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9628.136515] in-flight: 647:recv_work [ 9628.136524] workqueue ptlrpc_pinger: flags=0x2 [ 9628.136526] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9628.164396] ORIG_RAX: ffffffffffffff13 [ 9628.167020] pending: ptlrpc_pinger_main [ptlrpc] [ 9628.167497] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9618s workers=2 idle: 652 [ 9628.438890] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9628.443967] RDX: 00000000000013ae RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9628.448669] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9628.452752] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9628.456262] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9628.468638] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9628.474420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9628.477165] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9628.481756] Call Trace: [ 9628.485233] [ 9628.486893] ? show_regs.cold.9+0x22/0x2f [ 9628.497025] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9628.503276] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9628.513678] ? watchdog+0x40/0x40 [ 9628.516997] ? hrtimer_interrupt+0x128/0x2c0 [ 9628.519517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9628.523253] ? apic_timer_interrupt+0xf/0x20 [ 9628.528500] [ 9628.529891] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9628.534244] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9628.538917] do_raw_spin_lock+0x177/0x190 [ 9628.542958] _raw_spin_lock+0x21/0x30 [ 9628.545887] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9628.549507] obd_mod_free+0x46/0x140 [ptlrpc] [ 9628.552442] mdc_commit_open+0x91/0xb0 [mdc] [ 9628.557517] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9628.561497] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9628.567135] after_reply+0xe2c/0x1780 [ptlrpc] [ 9628.570637] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9628.579194] ? __next_timer_interrupt+0x160/0x160 [ 9628.583496] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9628.587963] ? do_wait_intr+0xf0/0xf0 [ 9628.590131] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9628.597906] kthread+0x1d1/0x200 [ 9628.602277] ? set_kthread_struct+0x70/0x70 [ 9628.604933] ret_from_fork+0x1f/0x30 [ 9642.608062] rcu: INFO: rcu_sched self-detected stall on CPU [ 9642.614156] rcu: 3-...!: (4822834 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [ 9642.623934] (t=7620217 jiffies g=269633 q=2701) [ 9642.629664] rcu: rcu_sched kthread starved for 7619869 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 9642.636788] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9642.647218] rcu: RCU grace-period kthread stack dump: [ 9642.652000] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9642.658828] Call Trace: [ 9642.659623] __schedule+0x351/0xcb0 [ 9642.661522] schedule+0xc0/0x180 [ 9642.663771] schedule_timeout+0xb4/0x190 [ 9642.665912] ? do_raw_spin_unlock+0x75/0x190 [ 9642.667969] ? __next_timer_interrupt+0x160/0x160 [ 9642.670629] ? prepare_to_swait_event+0xf3/0x140 [ 9642.672847] rcu_gp_kthread+0x8dd/0x15b0 [ 9642.677637] ? force_qs_rnp+0x2c0/0x2c0 [ 9642.681079] kthread+0x1d1/0x200 [ 9642.683969] ? set_kthread_struct+0x70/0x70 [ 9642.686718] ret_from_fork+0x1f/0x30 [ 9642.688318] rcu: Stack dump where RCU GP kthread last ran: [ 9642.694782] Sending NMI from CPU 3 to CPUs 2: [ 9642.702554] NMI backtrace for cpu 2 [ 9642.702561] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9642.702563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9642.702564] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9642.702566] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9642.702568] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 [ 9642.702571] RAX: 0000000000002c5b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9642.702572] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9642.702573] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9642.702575] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9642.702576] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9642.702577] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9642.702579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9642.702580] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9642.702581] Call Trace: [ 9642.702582] [ 9642.702583] ? show_regs.cold.9+0x22/0x2f [ 9642.702584] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9642.702586] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9642.702587] ? nmi_handle+0xbd/0x210 [ 9642.702588] ? default_do_nmi+0x4b/0x1b0 [ 9642.702589] ? do_nmi+0x280/0x380 [ 9642.702590] ? end_repeat_nmi+0x16/0x1f [ 9642.702592] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9642.702593] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9642.702594] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9642.702596] [ 9642.702597] do_raw_spin_lock+0x177/0x190 [ 9642.702598] _raw_spin_lock+0x21/0x30 [ 9642.702599] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9642.702600] ? do_raw_spin_unlock+0x75/0x190 [ 9642.702601] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9642.702603] ? do_wait_intr+0xf0/0xf0 [ 9642.702604] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9642.702605] mdc_close+0x53b/0xd70 [mdc] [ 9642.702606] lmv_close+0x18d/0x490 [lmv] [ 9642.702607] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9642.702608] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9642.702609] ll_md_close+0x274/0x800 [lustre] [ 9642.702610] ll_file_release+0x128/0xc50 [lustre] [ 9642.702611] __fput+0x10d/0x440 [ 9642.702612] ____fput+0x12/0x20 [ 9642.702613] task_work_run+0xc6/0x110 [ 9642.702615] exit_to_usermode_loop+0x1dd/0x1f0 [ 9642.702616] do_syscall_64+0x3d6/0x440 [ 9642.702617] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9642.702618] RIP: 0033:0x7f1a63983cd8 [ 9642.702620] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9642.702622] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9642.702624] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9642.702625] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9642.702626] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9642.702628] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9642.702629] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9642.704857] NMI backtrace for cpu 3 [ 9642.946090] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9642.954117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9642.967823] Call Trace: [ 9642.973666] [ 9642.978473] ? dump_stack+0xbb/0x10e [ 9642.982915] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9642.988249] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9642.991689] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9642.994900] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9643.003355] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9643.008018] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9643.010127] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9643.020435] ? update_process_times+0x6d/0xc0 [ 9643.022979] ? tick_sched_handle.isra.18+0x27/0x90 [ 9643.026803] ? tick_sched_timer+0x53/0xb0 [ 9643.028197] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9643.034548] ? tick_sched_handle.isra.18+0x90/0x90 [ 9643.036027] ? hrtimer_interrupt+0x128/0x2c0 [ 9643.040160] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9643.043854] ? apic_timer_interrupt+0xf/0x20 [ 9643.051596] [ 9643.052345] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9643.059936] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9643.065518] ? do_raw_spin_lock+0x177/0x190 [ 9643.068010] ? _raw_spin_lock+0x21/0x30 [ 9643.073557] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9643.082655] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9643.084646] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9643.091890] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9643.098208] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9643.103072] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9643.105919] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9643.109952] ? __next_timer_interrupt+0x160/0x160 [ 9643.113595] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9643.116065] ? do_wait_intr+0xf0/0xf0 [ 9643.117845] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9643.120626] ? kthread+0x1d1/0x200 [ 9643.122514] ? set_kthread_struct+0x70/0x70 [ 9643.124222] ? ret_from_fork+0x1f/0x30 [ 9648.056868] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9648.063748] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9648.104731] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9648.114737] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9648.122354] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9648.128262] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9648.145669] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9648.155102] RAX: 0000000000004b83 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9648.161911] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9648.165190] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9648.173239] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9648.182592] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9648.189164] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9648.196280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9648.203589] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9648.209792] Call Trace: [ 9648.214515] [ 9648.215374] ? show_regs.cold.9+0x22/0x2f [ 9648.217885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9648.222906] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9648.228530] ? watchdog+0x40/0x40 [ 9648.229726] ? hrtimer_interrupt+0x128/0x2c0 [ 9648.233100] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9648.238956] ? apic_timer_interrupt+0xf/0x20 [ 9648.243980] [ 9648.245786] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9648.248921] do_raw_spin_lock+0x177/0x190 [ 9648.254351] _raw_spin_lock+0x21/0x30 [ 9648.255878] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9648.262000] ? do_raw_spin_unlock+0x75/0x190 [ 9648.266885] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9648.270737] ? do_wait_intr+0xf0/0xf0 [ 9648.275614] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9648.277949] mdc_close+0x53b/0xd70 [mdc] [ 9648.283053] lmv_close+0x18d/0x490 [lmv] [ 9648.286017] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9648.291909] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9648.298530] ll_md_close+0x274/0x800 [lustre] [ 9648.300740] ll_file_release+0x128/0xc50 [lustre] [ 9648.306527] __fput+0x10d/0x440 [ 9648.308578] ____fput+0x12/0x20 [ 9648.312547] task_work_run+0xc6/0x110 [ 9648.317718] exit_to_usermode_loop+0x1dd/0x1f0 [ 9648.319538] do_syscall_64+0x3d6/0x440 [ 9648.321122] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9648.327328] RIP: 0033:0x7f1a63983cd8 [ 9648.329178] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9648.346755] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9648.351792] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9648.358580] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9648.368161] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9648.372806] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9648.378309] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9656.053960] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9656.058878] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9656.090839] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9656.103120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9656.122989] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9656.128889] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9656.150975] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9656.158326] RAX: 0000000000004284 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9656.163679] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9656.170856] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9656.175948] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9656.179492] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9656.184247] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9656.189816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9656.194613] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9656.197458] Call Trace: [ 9656.201490] [ 9656.204152] ? show_regs.cold.9+0x22/0x2f [ 9656.206020] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9656.211255] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9656.214529] ? watchdog+0x40/0x40 [ 9656.215899] ? hrtimer_interrupt+0x128/0x2c0 [ 9656.227034] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9656.246184] ? apic_timer_interrupt+0xf/0x20 [ 9656.251254] [ 9656.253820] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9656.259679] do_raw_spin_lock+0x177/0x190 [ 9656.269922] _raw_spin_lock+0x21/0x30 [ 9656.277596] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9656.281620] ? do_raw_spin_unlock+0x75/0x190 [ 9656.285408] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9656.292426] ? do_wait_intr+0xf0/0xf0 [ 9656.299131] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9656.304593] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9656.310917] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9656.317206] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9656.323948] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9656.336658] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9656.352022] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9656.367054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9656.373143] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9656.379633] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9656.381707] lmv_intent_open+0x338/0x1150 [lmv] [ 9656.396317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9656.402249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9656.406125] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9656.414315] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9656.425444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9656.434930] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9656.438102] ? do_raw_spin_unlock+0x75/0x190 [ 9656.471736] ? _raw_spin_unlock+0x12/0x30 [ 9656.475473] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9656.480145] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9656.484836] ll_atomic_open+0x36a/0xd70 [lustre] [ 9656.486634] lookup_open+0x356/0xb40 [ 9656.493103] do_last+0x313/0xfc0 [ 9656.495250] ? nd_jump_root+0xe5/0x160 [ 9656.499179] ? path_init+0x437/0x520 [ 9656.508020] path_openat+0xf7/0x500 [ 9656.516393] ? do_raw_spin_unlock+0x75/0x190 [ 9656.521592] do_filp_open+0x99/0x140 [ 9656.530228] ? getname_flags+0x6e/0x330 [ 9656.536868] ? __check_object_size+0xff/0x256 [ 9656.550442] ? do_raw_spin_unlock+0x75/0x190 [ 9656.551945] ? _raw_spin_unlock+0x12/0x30 [ 9656.561223] do_sys_openat2+0x2b4/0x410 [ 9656.572244] do_sys_open+0x73/0xa0 [ 9656.577420] __x64_sys_openat+0x24/0x30 [ 9656.578770] do_syscall_64+0xc1/0x440 [ 9656.590860] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9656.606664] RIP: 0033:0x7f1a63983332 [ 9656.607968] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9656.666702] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9656.685359] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9656.706665] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9656.717142] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9656.740208] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9656.763168] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9658.341050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7635s! [ 9658.343785] Showing busy workqueues and worker pools: [ 9658.345689] workqueue events: flags=0x0 [ 9658.347626] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9658.350588] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9658.354605] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9658.358284] pending: sec_gc_main [ptlrpc] [ 9658.360558] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9658.363818] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9658.368290] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9658.375074] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9658.379823] workqueue events_long: flags=0x0 [ 9658.382110] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9658.385544] pending: lu_objects_flush [obdclass] [ 9658.389595] workqueue events_unbound: flags=0x2 [ 9658.391748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9658.394782] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9658.403744] workqueue events_power_efficient: flags=0x80 [ 9658.409257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9658.412692] pending: neigh_periodic_work, do_cache_clean [ 9658.415793] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9658.418721] pending: neigh_periodic_work, check_lifetime [ 9658.422022] workqueue mm_percpu_wq: flags=0x8 [ 9658.424352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9658.427624] pending: vmstat_update [ 9658.429098] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9658.431996] pending: vmstat_update [ 9658.433596] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9658.436591] pending: vmstat_update [ 9658.438440] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9658.441632] pending: vmstat_update [ 9658.443257] workqueue writeback: flags=0x4a [ 9658.445135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9658.447731] pending: wb_workfn, wb_workfn [ 9658.451172] workqueue xprtiod: flags=0xa [ 9658.454277] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9658.460971] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9658.466566] workqueue ipv6_addrconf: flags=0x40008 [ 9658.470039] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9658.477837] pending: addrconf_verify_work [ 9658.479651] workqueue knbd0-recv: flags=0x1a [ 9658.481785] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9658.484968] in-flight: 647:recv_work [ 9658.486474] workqueue ptlrpc_pinger: flags=0x2 [ 9658.488577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9658.491259] pending: ptlrpc_pinger_main [ptlrpc] [ 9658.493834] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9648s workers=2 idle: 652 [ 9668.057047] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 9668.058068] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 9668.065295] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9668.065432] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9668.065438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9668.065441] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9668.065455] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 9668.065459] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9668.065463] RAX: 000000000000068a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9668.065466] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9668.065468] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9668.065470] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9668.065472] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9668.065475] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9668.065477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9668.065480] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9668.065488] Call Trace: [ 9668.066491] [ 9668.066510] ? show_regs.cold.9+0x22/0x2f [ 9668.066523] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9668.066530] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9668.066534] ? watchdog+0x40/0x40 [ 9668.066539] ? hrtimer_interrupt+0x128/0x2c0 [ 9668.066543] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9668.066552] ? apic_timer_interrupt+0xf/0x20 [ 9668.066563] [ 9668.066566] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9668.066573] do_raw_spin_lock+0x177/0x190 [ 9668.066578] _raw_spin_lock+0x21/0x30 [ 9668.066583] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9668.066856] ? do_raw_spin_unlock+0x75/0x190 [ 9668.066862] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9668.067855] ? do_wait_intr+0xf0/0xf0 [ 9668.067864] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9668.068362] mdc_close+0x53b/0xd70 [mdc] [ 9668.068399] lmv_close+0x18d/0x490 [lmv] [ 9668.068429] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9668.068748] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9668.068832] ll_md_close+0x274/0x800 [lustre] [ 9668.068912] ll_file_release+0x128/0xc50 [lustre] [ 9668.069083] __fput+0x10d/0x440 [ 9668.069092] ____fput+0x12/0x20 [ 9668.069097] task_work_run+0xc6/0x110 [ 9668.069104] exit_to_usermode_loop+0x1dd/0x1f0 [ 9668.069111] do_syscall_64+0x3d6/0x440 [ 9668.069116] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9668.069123] RIP: 0033:0x7f1a63983cd8 [ 9668.069132] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9668.074886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9668.075089] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9668.075096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9668.075099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9668.075111] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9668.075115] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9668.075120] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9668.075122] RDX: 00000000000057d5 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9668.075124] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9668.075127] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9668.075134] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9668.075137] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9668.075140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9668.075209] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9668.075220] Call Trace: [ 9668.075226] [ 9668.075232] ? show_regs.cold.9+0x22/0x2f [ 9668.075242] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9668.075249] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9668.075254] ? watchdog+0x40/0x40 [ 9668.075258] ? hrtimer_interrupt+0x128/0x2c0 [ 9668.075262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9668.075271] ? apic_timer_interrupt+0xf/0x20 [ 9668.075275] [ 9668.075278] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9668.075283] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9668.075287] do_raw_spin_lock+0x177/0x190 [ 9668.075292] _raw_spin_lock+0x21/0x30 [ 9668.075297] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9668.111383] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9668.111399] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9668.111401] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9668.111404] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9668.111406] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9668.111409] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9668.605897] obd_mod_free+0x46/0x140 [ptlrpc] [ 9668.608212] mdc_commit_open+0x91/0xb0 [mdc] [ 9668.610189] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9668.613986] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9668.621624] after_reply+0xe2c/0x1780 [ptlrpc] [ 9668.623580] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9668.630850] ? __next_timer_interrupt+0x160/0x160 [ 9668.633821] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9668.642105] ? do_wait_intr+0xf0/0xf0 [ 9668.649950] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9668.654649] kthread+0x1d1/0x200 [ 9668.658554] ? set_kthread_struct+0x70/0x70 [ 9668.660590] ret_from_fork+0x1f/0x30 [ 9676.056645] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9676.067635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9676.132621] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9676.142024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9676.151166] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9676.155572] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9676.180433] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9676.187140] RAX: 0000000000007b85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9676.193745] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9676.202943] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9676.208031] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9676.217755] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9676.225830] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9676.241479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9676.249138] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9676.255599] Call Trace: [ 9676.257420] [ 9676.259647] ? show_regs.cold.9+0x22/0x2f [ 9676.262208] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9676.268626] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9676.275211] ? watchdog+0x40/0x40 [ 9676.277623] ? hrtimer_interrupt+0x128/0x2c0 [ 9676.283983] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9676.288453] ? apic_timer_interrupt+0xf/0x20 [ 9676.291153] [ 9676.292360] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9676.300909] do_raw_spin_lock+0x177/0x190 [ 9676.303763] _raw_spin_lock+0x21/0x30 [ 9676.309195] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9676.316752] ? do_raw_spin_unlock+0x75/0x190 [ 9676.320872] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9676.326143] ? do_wait_intr+0xf0/0xf0 [ 9676.328019] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9676.346402] mdc_close+0x53b/0xd70 [mdc] [ 9676.355747] lmv_close+0x18d/0x490 [lmv] [ 9676.368780] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9676.375022] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9676.389940] ll_md_close+0x274/0x800 [lustre] [ 9676.398925] ll_file_release+0x128/0xc50 [lustre] [ 9676.410665] __fput+0x10d/0x440 [ 9676.415369] ____fput+0x12/0x20 [ 9676.419813] task_work_run+0xc6/0x110 [ 9676.428827] exit_to_usermode_loop+0x1dd/0x1f0 [ 9676.431057] do_syscall_64+0x3d6/0x440 [ 9676.434715] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9676.441871] RIP: 0033:0x7f1a63983cd8 [ 9676.447544] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9676.462992] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9676.467516] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9676.476247] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9676.484955] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9676.495412] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9676.499828] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9684.053031] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9684.056354] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9684.077294] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9684.082690] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9684.087258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9684.090004] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9684.099797] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9684.103554] RAX: 000000000000748e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9684.107025] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9684.111251] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9684.114601] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9684.117611] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9684.119815] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9684.123826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9684.126902] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9684.130500] Call Trace: [ 9684.132081] [ 9684.133140] ? show_regs.cold.9+0x22/0x2f [ 9684.134938] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9684.137181] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9684.139256] ? watchdog+0x40/0x40 [ 9684.140361] ? hrtimer_interrupt+0x128/0x2c0 [ 9684.142054] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9684.144257] ? apic_timer_interrupt+0xf/0x20 [ 9684.146255] [ 9684.147092] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9684.149606] do_raw_spin_lock+0x177/0x190 [ 9684.151505] _raw_spin_lock+0x21/0x30 [ 9684.152884] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9684.155445] ? do_raw_spin_unlock+0x75/0x190 [ 9684.157645] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9684.159691] ? do_wait_intr+0xf0/0xf0 [ 9684.160953] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9684.162736] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9684.164550] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9684.166613] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9684.168380] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9684.169940] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9684.171607] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9684.172940] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9684.174782] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9684.176706] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9684.178546] lmv_intent_open+0x338/0x1150 [lmv] [ 9684.180240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9684.182617] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9684.184761] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9684.186594] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9684.188631] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9684.191489] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9684.193658] ? do_raw_spin_unlock+0x75/0x190 [ 9684.195667] ? _raw_spin_unlock+0x12/0x30 [ 9684.200830] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9684.203616] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9684.206041] ll_atomic_open+0x36a/0xd70 [lustre] [ 9684.210587] lookup_open+0x356/0xb40 [ 9684.215927] do_last+0x313/0xfc0 [ 9684.217177] ? nd_jump_root+0xe5/0x160 [ 9684.223402] ? path_init+0x437/0x520 [ 9684.225126] path_openat+0xf7/0x500 [ 9684.226347] ? do_raw_spin_unlock+0x75/0x190 [ 9684.239213] do_filp_open+0x99/0x140 [ 9684.242855] ? getname_flags+0x6e/0x330 [ 9684.245621] ? __check_object_size+0xff/0x256 [ 9684.247669] ? do_raw_spin_unlock+0x75/0x190 [ 9684.251215] ? _raw_spin_unlock+0x12/0x30 [ 9684.254724] do_sys_openat2+0x2b4/0x410 [ 9684.256926] do_sys_open+0x73/0xa0 [ 9684.259813] __x64_sys_openat+0x24/0x30 [ 9684.261678] do_syscall_64+0xc1/0x440 [ 9684.263812] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9684.266825] RIP: 0033:0x7f1a63983332 [ 9684.267855] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9684.282897] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9684.287956] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9684.291107] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9684.294126] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9684.297356] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9684.300873] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9688.545798] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7665s! [ 9688.549013] Showing busy workqueues and worker pools: [ 9688.564688] workqueue events: flags=0x0 [ 9688.568783] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9688.585643] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9688.594763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9688.602493] pending: sec_gc_main [ptlrpc] [ 9688.607429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9688.612931] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9688.621667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9688.625192] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9688.632347] workqueue events_long: flags=0x0 [ 9688.637429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9688.648699] pending: lu_objects_flush [obdclass] [ 9688.653905] workqueue events_unbound: flags=0x2 [ 9688.664178] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9688.678216] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9688.694921] workqueue events_power_efficient: flags=0x80 [ 9688.700608] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9688.706721] pending: neigh_periodic_work, do_cache_clean [ 9688.714105] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9688.718714] pending: neigh_periodic_work, check_lifetime [ 9688.723240] workqueue mm_percpu_wq: flags=0x8 [ 9688.725813] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9688.730286] pending: vmstat_update [ 9688.735590] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9688.742902] pending: vmstat_update [ 9688.743984] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9688.749744] pending: vmstat_update [ 9688.751356] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9688.757503] pending: vmstat_update [ 9688.758644] workqueue writeback: flags=0x4a [ 9688.763763] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9688.768183] pending: wb_workfn, wb_workfn [ 9688.771161] workqueue xprtiod: flags=0xa [ 9688.775612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9688.780625] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9688.787780] workqueue ipv6_addrconf: flags=0x40008 [ 9688.794704] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9688.803009] pending: addrconf_verify_work [ 9688.804504] workqueue knbd0-recv: flags=0x1a [ 9688.811215] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9688.817612] in-flight: 647:recv_work [ 9688.819846] workqueue ptlrpc_pinger: flags=0x2 [ 9688.825407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9688.827414] pending: ptlrpc_pinger_main [ptlrpc] [ 9688.837567] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9678s workers=2 idle: 652 [ 9696.060920] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 9696.061283] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 9696.071307] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9696.071395] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9696.071403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9696.071407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9696.071421] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9696.071424] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9696.071429] RAX: 00000000000069c9 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9696.071431] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9696.071433] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9696.071435] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9696.071437] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9696.071440] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9696.071443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9696.071445] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9696.071456] Call Trace: [ 9696.071461] [ 9696.071467] ? show_regs.cold.9+0x22/0x2f [ 9696.071477] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9696.071482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9696.071487] ? watchdog+0x40/0x40 [ 9696.071491] ? hrtimer_interrupt+0x128/0x2c0 [ 9696.071495] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9696.071503] ? apic_timer_interrupt+0xf/0x20 [ 9696.071508] [ 9696.071511] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9696.071516] do_raw_spin_lock+0x177/0x190 [ 9696.071521] _raw_spin_lock+0x21/0x30 [ 9696.071526] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9696.124227] Modules linked in: [ 9696.149326] ? do_raw_spin_unlock+0x75/0x190 [ 9696.150846] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9696.162848] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9696.162937] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9696.162944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9696.162947] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9696.162961] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9696.162965] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9696.162969] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9696.162971] RDX: 0000000000002167 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9696.162974] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9696.162976] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9696.162978] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9696.162981] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9696.162983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9696.162985] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9696.162993] Call Trace: [ 9696.162998] [ 9696.163003] ? show_regs.cold.9+0x22/0x2f [ 9696.163011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9696.163018] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9696.163022] ? watchdog+0x40/0x40 [ 9696.163027] ? hrtimer_interrupt+0x128/0x2c0 [ 9696.163030] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9696.163038] ? apic_timer_interrupt+0xf/0x20 [ 9696.163042] [ 9696.163045] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9696.163049] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9696.163052] do_raw_spin_lock+0x177/0x190 [ 9696.163056] _raw_spin_lock+0x21/0x30 [ 9696.163061] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9696.163325] obd_mod_free+0x46/0x140 [ptlrpc] [ 9696.163779] mdc_commit_open+0x91/0xb0 [mdc] [ 9696.163816] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9696.164082] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9696.164401] after_reply+0xe2c/0x1780 [ptlrpc] [ 9696.164997] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9696.165195] ? __next_timer_interrupt+0x160/0x160 [ 9696.165207] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9696.165430] ? do_wait_intr+0xf0/0xf0 [ 9696.165440] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9696.165728] kthread+0x1d1/0x200 [ 9696.165734] ? set_kthread_struct+0x70/0x70 [ 9696.165738] ret_from_fork+0x1f/0x30 [ 9696.619372] ? do_wait_intr+0xf0/0xf0 [ 9696.623320] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9696.630652] mdc_close+0x53b/0xd70 [mdc] [ 9696.631698] lmv_close+0x18d/0x490 [lmv] [ 9696.636148] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9696.637733] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9696.640533] ll_md_close+0x274/0x800 [lustre] [ 9696.643036] ll_file_release+0x128/0xc50 [lustre] [ 9696.646068] __fput+0x10d/0x440 [ 9696.651000] ____fput+0x12/0x20 [ 9696.653310] task_work_run+0xc6/0x110 [ 9696.655129] exit_to_usermode_loop+0x1dd/0x1f0 [ 9696.659512] do_syscall_64+0x3d6/0x440 [ 9696.664594] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9696.666466] RIP: 0033:0x7f1a63983cd8 [ 9696.681567] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9696.720049] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9696.737210] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9696.746528] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9696.750332] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9696.761844] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9696.774643] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9704.056025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9704.059100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9704.087178] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9704.092455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9704.099643] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9704.101755] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9704.120522] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9704.124778] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9704.127508] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9704.133171] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9704.139602] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9704.145008] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9704.149037] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9704.158603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9704.164597] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9704.172674] Call Trace: [ 9704.174703] [ 9704.176094] ? show_regs.cold.9+0x22/0x2f [ 9704.177870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9704.183434] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9704.186551] ? watchdog+0x40/0x40 [ 9704.188038] ? hrtimer_interrupt+0x128/0x2c0 [ 9704.190193] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9704.192147] ? apic_timer_interrupt+0xf/0x20 [ 9704.194371] [ 9704.195139] ? native_safe_halt+0x2e/0x50 [ 9704.196543] kvm_wait+0xcd/0xe0 [ 9704.197789] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9704.204569] do_raw_spin_lock+0x177/0x190 [ 9704.207229] _raw_spin_lock+0x21/0x30 [ 9704.212739] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9704.217389] ? do_raw_spin_unlock+0x75/0x190 [ 9704.219207] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9704.222237] ? do_wait_intr+0xf0/0xf0 [ 9704.223269] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9704.225297] mdc_close+0x53b/0xd70 [mdc] [ 9704.226538] lmv_close+0x18d/0x490 [lmv] [ 9704.228725] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9704.231642] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9704.233599] ll_md_close+0x274/0x800 [lustre] [ 9704.235790] ll_file_release+0x128/0xc50 [lustre] [ 9704.238582] __fput+0x10d/0x440 [ 9704.239746] ____fput+0x12/0x20 [ 9704.241410] task_work_run+0xc6/0x110 [ 9704.244643] exit_to_usermode_loop+0x1dd/0x1f0 [ 9704.246399] do_syscall_64+0x3d6/0x440 [ 9704.247787] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9704.256491] RIP: 0033:0x7f1a63983cd8 [ 9704.260425] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9704.271018] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9704.275205] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9704.278158] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9704.281453] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9704.287733] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9704.290452] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9716.053085] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9716.055614] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9716.088533] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9716.094092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9716.098098] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9716.105405] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9716.114464] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9716.117036] RAX: 0000000000006691 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9716.119810] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9716.123224] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9716.128249] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9716.133456] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9716.137000] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9716.140773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9716.144338] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9716.147852] Call Trace: [ 9716.149025] [ 9716.150000] ? show_regs.cold.9+0x22/0x2f [ 9716.154421] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9716.156244] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9716.158228] ? watchdog+0x40/0x40 [ 9716.159497] ? hrtimer_interrupt+0x128/0x2c0 [ 9716.161441] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9716.163447] ? apic_timer_interrupt+0xf/0x20 [ 9716.165242] [ 9716.166123] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9716.168496] do_raw_spin_lock+0x177/0x190 [ 9716.170138] _raw_spin_lock+0x21/0x30 [ 9716.171587] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9716.174433] ? do_raw_spin_unlock+0x75/0x190 [ 9716.178009] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9716.180578] ? do_wait_intr+0xf0/0xf0 [ 9716.183934] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9716.187197] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9716.190525] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9716.193654] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9716.196466] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9716.202740] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9716.213653] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9716.218050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9716.220986] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9716.230433] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9716.236676] lmv_intent_open+0x338/0x1150 [lmv] [ 9716.238640] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9716.246977] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9716.252941] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9716.259734] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9716.265406] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9716.269006] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9716.271342] ? do_raw_spin_unlock+0x75/0x190 [ 9716.273298] ? _raw_spin_unlock+0x12/0x30 [ 9716.275186] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9716.281530] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9716.287962] ll_atomic_open+0x36a/0xd70 [lustre] [ 9716.291456] lookup_open+0x356/0xb40 [ 9716.293432] do_last+0x313/0xfc0 [ 9716.297375] ? nd_jump_root+0xe5/0x160 [ 9716.299293] ? path_init+0x437/0x520 [ 9716.301114] path_openat+0xf7/0x500 [ 9716.302609] ? do_raw_spin_unlock+0x75/0x190 [ 9716.305112] do_filp_open+0x99/0x140 [ 9716.306561] ? getname_flags+0x6e/0x330 [ 9716.311359] ? __check_object_size+0xff/0x256 [ 9716.313316] ? do_raw_spin_unlock+0x75/0x190 [ 9716.315607] ? _raw_spin_unlock+0x12/0x30 [ 9716.319933] do_sys_openat2+0x2b4/0x410 [ 9716.322943] do_sys_open+0x73/0xa0 [ 9716.325081] __x64_sys_openat+0x24/0x30 [ 9716.326760] do_syscall_64+0xc1/0x440 [ 9716.328523] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9716.331131] RIP: 0033:0x7f1a63983332 [ 9716.332665] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9716.342748] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9716.346222] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9716.349544] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9716.355932] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9716.359846] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9716.364016] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9719.265042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7696s! [ 9719.269089] Showing busy workqueues and worker pools: [ 9719.271156] workqueue events: flags=0x0 [ 9719.272605] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9719.277255] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9719.283596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9719.290049] pending: sec_gc_main [ptlrpc] [ 9719.294030] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9719.298446] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9719.305041] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9719.307032] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9719.316143] workqueue events_long: flags=0x0 [ 9719.325026] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9719.333563] pending: lu_objects_flush [obdclass] [ 9719.344687] workqueue events_unbound: flags=0x2 [ 9719.350275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9719.357014] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9719.363931] workqueue events_power_efficient: flags=0x80 [ 9719.368452] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9719.375213] pending: neigh_periodic_work, do_cache_clean [ 9719.377880] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9719.388444] pending: neigh_periodic_work, check_lifetime [ 9719.393146] workqueue mm_percpu_wq: flags=0x8 [ 9719.403072] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9719.410611] pending: vmstat_update [ 9719.412913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9719.417126] pending: vmstat_update [ 9719.421177] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9719.429424] pending: vmstat_update [ 9719.434069] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9719.441159] pending: vmstat_update [ 9719.442552] workqueue writeback: flags=0x4a [ 9719.449284] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9719.457114] pending: wb_workfn, wb_workfn [ 9719.459208] workqueue xprtiod: flags=0xa [ 9719.462036] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9719.465031] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9719.468872] workqueue ipv6_addrconf: flags=0x40008 [ 9719.470970] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9719.473898] pending: addrconf_verify_work [ 9719.475535] workqueue knbd0-recv: flags=0x1a [ 9719.480454] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9719.483281] in-flight: 647:recv_work [ 9719.489128] workqueue ptlrpc_pinger: flags=0x2 [ 9719.491486] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9719.495456] pending: ptlrpc_pinger_main [ptlrpc] [ 9719.507305] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9709s workers=2 idle: 652 [ 9724.057029] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 9724.062587] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 9724.069213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9724.070041] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9724.070049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9724.070053] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9724.070067] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9724.070073] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9724.070078] RAX: 000000000000700a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9724.070081] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9724.070083] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9724.070086] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9724.070088] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9724.070092] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9724.070095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9724.070098] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9724.070106] Call Trace: [ 9724.070112] [ 9724.070118] ? show_regs.cold.9+0x22/0x2f [ 9724.070128] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9724.070136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9724.070140] ? watchdog+0x40/0x40 [ 9724.070145] ? hrtimer_interrupt+0x128/0x2c0 [ 9724.070150] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9724.070160] ? apic_timer_interrupt+0xf/0x20 [ 9724.070164] [ 9724.070166] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9724.070172] do_raw_spin_lock+0x177/0x190 [ 9724.070176] _raw_spin_lock+0x21/0x30 [ 9724.070180] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9724.076664] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9724.076744] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9724.076752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9724.076755] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9724.076770] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9724.076774] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9724.076778] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9724.076781] RDX: 0000000000002148 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9724.076782] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9724.076785] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9724.076787] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9724.076790] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9724.076793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9724.076795] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9724.076804] Call Trace: [ 9724.076810] [ 9724.076815] ? show_regs.cold.9+0x22/0x2f [ 9724.076824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9724.076830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9724.076835] ? watchdog+0x40/0x40 [ 9724.076839] ? hrtimer_interrupt+0x128/0x2c0 [ 9724.076843] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9724.076851] ? apic_timer_interrupt+0xf/0x20 [ 9724.076855] [ 9724.076857] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9724.076868] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9724.078002] do_raw_spin_lock+0x177/0x190 [ 9724.078019] _raw_spin_lock+0x21/0x30 [ 9724.078026] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9724.078326] obd_mod_free+0x46/0x140 [ptlrpc] [ 9724.078519] mdc_commit_open+0x91/0xb0 [mdc] [ 9724.078555] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9724.078768] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9724.080647] after_reply+0xe2c/0x1780 [ptlrpc] [ 9724.114917] ? do_raw_spin_unlock+0x75/0x190 [ 9724.114942] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9724.121865] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9724.122278] ? __next_timer_interrupt+0x160/0x160 [ 9724.122290] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9724.122697] ? do_wait_intr+0xf0/0xf0 [ 9724.122708] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9724.122940] kthread+0x1d1/0x200 [ 9724.122946] ? set_kthread_struct+0x70/0x70 [ 9724.122950] ret_from_fork+0x1f/0x30 [ 9724.374212] ? do_wait_intr+0xf0/0xf0 [ 9724.376163] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9724.378961] mdc_close+0x53b/0xd70 [mdc] [ 9724.381095] lmv_close+0x18d/0x490 [lmv] [ 9724.383104] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9724.386065] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9724.388732] ll_md_close+0x274/0x800 [lustre] [ 9724.391211] ll_file_release+0x128/0xc50 [lustre] [ 9724.393844] __fput+0x10d/0x440 [ 9724.395350] ____fput+0x12/0x20 [ 9724.397138] task_work_run+0xc6/0x110 [ 9724.398991] exit_to_usermode_loop+0x1dd/0x1f0 [ 9724.401301] do_syscall_64+0x3d6/0x440 [ 9724.403216] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9724.405889] RIP: 0033:0x7f1a63983cd8 [ 9724.407819] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9724.417378] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9724.421252] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9724.424969] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9724.428864] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9724.432689] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9724.437878] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9732.056026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9732.058645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9732.076896] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9732.081633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9732.086307] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9732.091389] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9732.102485] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9732.106250] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9732.109390] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9732.113195] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9732.116792] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9732.120150] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9732.123306] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9732.126388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9732.127984] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9732.130454] Call Trace: [ 9732.131340] [ 9732.132134] ? show_regs.cold.9+0x22/0x2f [ 9732.133840] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9732.135520] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9732.137137] ? watchdog+0x40/0x40 [ 9732.138391] ? hrtimer_interrupt+0x128/0x2c0 [ 9732.139735] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9732.141440] ? apic_timer_interrupt+0xf/0x20 [ 9732.142867] [ 9732.143489] ? native_safe_halt+0x2e/0x50 [ 9732.145172] kvm_wait+0xcd/0xe0 [ 9732.146865] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9732.148756] do_raw_spin_lock+0x177/0x190 [ 9732.150213] _raw_spin_lock+0x21/0x30 [ 9732.151489] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9732.153868] ? do_raw_spin_unlock+0x75/0x190 [ 9732.155395] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9732.157511] ? do_wait_intr+0xf0/0xf0 [ 9732.158964] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9732.161331] mdc_close+0x53b/0xd70 [mdc] [ 9732.162937] lmv_close+0x18d/0x490 [lmv] [ 9732.164175] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9732.166111] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9732.167862] ll_md_close+0x274/0x800 [lustre] [ 9732.169385] ll_file_release+0x128/0xc50 [lustre] [ 9732.171071] __fput+0x10d/0x440 [ 9732.172049] ____fput+0x12/0x20 [ 9732.173867] task_work_run+0xc6/0x110 [ 9732.175011] exit_to_usermode_loop+0x1dd/0x1f0 [ 9732.176285] do_syscall_64+0x3d6/0x440 [ 9732.178063] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9732.179905] RIP: 0033:0x7f1a63983cd8 [ 9732.181442] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9732.191465] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9732.195364] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9732.201078] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9732.207193] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9732.210587] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9732.213465] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9744.053052] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9744.058818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9744.122818] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9744.139315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9744.149295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9744.153170] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9744.178344] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9744.184913] RAX: 0000000000003c74 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9744.190971] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9744.200084] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9744.203152] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9744.207343] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9744.213257] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9744.222093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9744.226236] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9744.234043] Call Trace: [ 9744.236829] [ 9744.237753] ? show_regs.cold.9+0x22/0x2f [ 9744.240245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9744.244307] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9744.248874] ? watchdog+0x40/0x40 [ 9744.254362] ? hrtimer_interrupt+0x128/0x2c0 [ 9744.258266] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9744.261920] ? apic_timer_interrupt+0xf/0x20 [ 9744.263973] [ 9744.264985] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9744.267387] do_raw_spin_lock+0x177/0x190 [ 9744.273045] _raw_spin_lock+0x21/0x30 [ 9744.274338] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9744.278045] ? do_raw_spin_unlock+0x75/0x190 [ 9744.280756] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9744.282807] ? do_wait_intr+0xf0/0xf0 [ 9744.284168] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9744.293373] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9744.296220] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9744.298958] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9744.301235] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9744.308189] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9744.310089] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9744.311548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9744.324509] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9744.332029] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9744.339335] lmv_intent_open+0x338/0x1150 [lmv] [ 9744.347268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9744.356650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9744.367616] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9744.370847] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9744.373971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9744.376859] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9744.379831] ? do_raw_spin_unlock+0x75/0x190 [ 9744.382324] ? _raw_spin_unlock+0x12/0x30 [ 9744.384265] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9744.393494] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9744.400348] ll_atomic_open+0x36a/0xd70 [lustre] [ 9744.406178] lookup_open+0x356/0xb40 [ 9744.409488] do_last+0x313/0xfc0 [ 9744.411161] ? nd_jump_root+0xe5/0x160 [ 9744.412913] ? path_init+0x437/0x520 [ 9744.414865] path_openat+0xf7/0x500 [ 9744.416352] ? do_raw_spin_unlock+0x75/0x190 [ 9744.424474] do_filp_open+0x99/0x140 [ 9744.428171] ? getname_flags+0x6e/0x330 [ 9744.431807] ? __check_object_size+0xff/0x256 [ 9744.434144] ? do_raw_spin_unlock+0x75/0x190 [ 9744.436344] ? _raw_spin_unlock+0x12/0x30 [ 9744.439602] do_sys_openat2+0x2b4/0x410 [ 9744.440894] do_sys_open+0x73/0xa0 [ 9744.441853] __x64_sys_openat+0x24/0x30 [ 9744.445065] do_syscall_64+0xc1/0x440 [ 9744.450196] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9744.458209] RIP: 0033:0x7f1a63983332 [ 9744.464849] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9744.478331] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9744.482609] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9744.495620] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9744.498157] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9744.507943] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9744.519001] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9749.986094] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7726s! [ 9749.990673] Showing busy workqueues and worker pools: [ 9749.994313] workqueue events: flags=0x0 [ 9750.000085] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9750.008119] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9750.015008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9750.020793] pending: sec_gc_main [ptlrpc] [ 9750.025158] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9750.030527] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9750.038628] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9750.042503] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9750.046150] workqueue events_long: flags=0x0 [ 9750.047936] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9750.053739] pending: lu_objects_flush [obdclass] [ 9750.056342] workqueue events_unbound: flags=0x2 [ 9750.059402] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9750.062616] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9750.071313] workqueue events_power_efficient: flags=0x80 [ 9750.074970] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9750.077987] pending: neigh_periodic_work, do_cache_clean [ 9750.082014] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9750.085316] pending: neigh_periodic_work, check_lifetime [ 9750.089032] workqueue mm_percpu_wq: flags=0x8 [ 9750.090797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9750.095023] pending: vmstat_update [ 9750.097126] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9750.100166] pending: vmstat_update [ 9750.106830] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9750.109024] pending: vmstat_update [ 9750.109981] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9750.114982] pending: vmstat_update [ 9750.116858] workqueue writeback: flags=0x4a [ 9750.119112] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9750.121913] pending: wb_workfn, wb_workfn [ 9750.124140] workqueue xprtiod: flags=0xa [ 9750.129977] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9750.132176] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9750.142873] workqueue ipv6_addrconf: flags=0x40008 [ 9750.145016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9750.147491] pending: addrconf_verify_work [ 9750.149760] workqueue knbd0-recv: flags=0x1a [ 9750.153384] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9750.155995] in-flight: 647:recv_work [ 9750.162069] workqueue ptlrpc_pinger: flags=0x2 [ 9750.168233] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9750.178657] pending: ptlrpc_pinger_main [ptlrpc] [ 9750.187083] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9740s workers=2 idle: 652 [ 9752.057557] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [ 9752.058026] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 9752.060122] Modules linked in: [ 9752.063281] Modules linked in: [ 9752.063284] lustre(O) osp(O) [ 9752.064587] lustre(O) [ 9752.065903] ofd(O) [ 9752.066895] osp(O) [ 9752.067914] lod(O) [ 9752.068918] ofd(O) [ 9752.069958] mdt(O) [ 9752.070917] lod(O) [ 9752.071907] mdd(O) [ 9752.072908] mdt(O) [ 9752.073967] mgs(O) [ 9752.074957] mdd(O) [ 9752.076221] lquota(O) [ 9752.077127] mgs(O) [ 9752.078187] lfsck(O) [ 9752.079248] lquota(O) [ 9752.080449] obdecho(O) [ 9752.081346] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9752.081409] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9752.081415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9752.081417] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9752.081431] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9752.081435] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9752.081440] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9752.081443] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9752.081444] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9752.081446] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9752.081448] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9752.081451] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9752.081453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9752.081455] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9752.081465] Call Trace: [ 9752.081471] [ 9752.081477] ? show_regs.cold.9+0x22/0x2f [ 9752.082467] mgc(O) [ 9752.083682] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9752.100770] mdc(O) [ 9752.105547] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9752.109899] lov(O) [ 9752.112108] ? watchdog+0x40/0x40 [ 9752.112132] ? hrtimer_interrupt+0x128/0x2c0 [ 9752.112143] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9752.112152] ? apic_timer_interrupt+0xf/0x20 [ 9752.112157] [ 9752.112162] ? native_safe_halt+0x2e/0x50 [ 9752.112167] kvm_wait+0xcd/0xe0 [ 9752.112177] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9752.112186] do_raw_spin_lock+0x177/0x190 [ 9752.112191] _raw_spin_lock+0x21/0x30 [ 9752.112196] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9752.112715] obd_mod_free+0x46/0x140 [ptlrpc] [ 9752.137596] osc(O) [ 9752.141945] mdc_commit_open+0x91/0xb0 [mdc] [ 9752.142009] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9752.156267] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9752.156328] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9752.156347] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9752.156350] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9752.156363] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9752.156368] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9752.156373] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9752.156375] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9752.156377] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9752.156379] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9752.156382] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9752.156386] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9752.156389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9752.156391] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9752.156398] Call Trace: [ 9752.156404] [ 9752.156410] ? show_regs.cold.9+0x22/0x2f [ 9752.156420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9752.156426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9752.156430] ? watchdog+0x40/0x40 [ 9752.156434] ? hrtimer_interrupt+0x128/0x2c0 [ 9752.156439] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9752.156445] ? apic_timer_interrupt+0xf/0x20 [ 9752.156449] [ 9752.156452] ? native_safe_halt+0x2e/0x50 [ 9752.156456] kvm_wait+0xcd/0xe0 [ 9752.156463] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9752.156471] do_raw_spin_lock+0x177/0x190 [ 9752.156476] _raw_spin_lock+0x21/0x30 [ 9752.156480] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9752.156756] ? do_raw_spin_unlock+0x75/0x190 [ 9752.156762] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9752.156955] ? do_wait_intr+0xf0/0xf0 [ 9752.156964] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9752.157161] mdc_close+0x53b/0xd70 [mdc] [ 9752.157203] lmv_close+0x18d/0x490 [lmv] [ 9752.157234] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9752.157354] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9752.161081] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9752.161759] after_reply+0xe2c/0x1780 [ptlrpc] [ 9752.162022] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9752.174805] ll_md_close+0x274/0x800 [lustre] [ 9752.178306] ? __next_timer_interrupt+0x160/0x160 [ 9752.178346] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9752.178881] ? do_wait_intr+0xf0/0xf0 [ 9752.178894] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9752.179151] kthread+0x1d1/0x200 [ 9752.179157] ? set_kthread_struct+0x70/0x70 [ 9752.179166] ret_from_fork+0x1f/0x30 [ 9752.514448] ll_file_release+0x128/0xc50 [lustre] [ 9752.519039] __fput+0x10d/0x440 [ 9752.522925] ____fput+0x12/0x20 [ 9752.523963] task_work_run+0xc6/0x110 [ 9752.527020] exit_to_usermode_loop+0x1dd/0x1f0 [ 9752.530532] do_syscall_64+0x3d6/0x440 [ 9752.533067] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9752.537144] RIP: 0033:0x7f1a63983cd8 [ 9752.538888] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9752.554954] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9752.558267] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9752.560973] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9752.569518] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9752.578924] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9752.589056] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9760.057983] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9760.061553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9760.128689] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9760.138836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9760.144971] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9760.148614] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9760.159941] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9760.163082] RAX: 0000000000000016 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9760.167083] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9760.171943] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9760.179046] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9760.187926] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9760.192448] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9760.196394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9760.199833] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9760.208672] Call Trace: [ 9760.213525] [ 9760.217314] ? show_regs.cold.9+0x22/0x2f [ 9760.222383] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9760.230119] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9760.237790] ? watchdog+0x40/0x40 [ 9760.245097] ? hrtimer_interrupt+0x128/0x2c0 [ 9760.252892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9760.259041] ? apic_timer_interrupt+0xf/0x20 [ 9760.264138] [ 9760.271341] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9760.276162] do_raw_spin_lock+0x177/0x190 [ 9760.280499] _raw_spin_lock+0x21/0x30 [ 9760.284669] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9760.292756] ? do_raw_spin_unlock+0x75/0x190 [ 9760.300625] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9760.311535] ? do_wait_intr+0xf0/0xf0 [ 9760.318144] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9760.327510] mdc_close+0x53b/0xd70 [mdc] [ 9760.334235] lmv_close+0x18d/0x490 [lmv] [ 9760.336788] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9760.345199] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9760.346964] ll_md_close+0x274/0x800 [lustre] [ 9760.357419] ll_file_release+0x128/0xc50 [lustre] [ 9760.367491] __fput+0x10d/0x440 [ 9760.374244] ____fput+0x12/0x20 [ 9760.377580] task_work_run+0xc6/0x110 [ 9760.383232] exit_to_usermode_loop+0x1dd/0x1f0 [ 9760.392503] do_syscall_64+0x3d6/0x440 [ 9760.397048] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9760.407463] RIP: 0033:0x7f1a63983cd8 [ 9760.416985] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9760.442489] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9760.456454] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9760.466121] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9760.476621] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9760.487193] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9760.505915] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9776.053026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9776.057566] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9776.090871] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9776.101777] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9776.108801] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9776.110964] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9776.124462] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9776.132657] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9776.137277] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9776.141705] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9776.145240] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9776.149768] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9776.159982] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9776.162643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9776.164649] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9776.168540] Call Trace: [ 9776.170033] [ 9776.171769] ? show_regs.cold.9+0x22/0x2f [ 9776.173487] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9776.174914] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9776.176832] ? watchdog+0x40/0x40 [ 9776.178556] ? hrtimer_interrupt+0x128/0x2c0 [ 9776.181833] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9776.184649] ? apic_timer_interrupt+0xf/0x20 [ 9776.188416] [ 9776.189585] ? native_safe_halt+0x2e/0x50 [ 9776.192176] kvm_wait+0xcd/0xe0 [ 9776.195436] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9776.204994] do_raw_spin_lock+0x177/0x190 [ 9776.207573] _raw_spin_lock+0x21/0x30 [ 9776.209329] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9776.213003] ? do_raw_spin_unlock+0x75/0x190 [ 9776.215848] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9776.218991] ? do_wait_intr+0xf0/0xf0 [ 9776.222413] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9776.224741] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9776.229724] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9776.235842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9776.240532] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9776.246017] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9776.250892] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9776.252888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9776.254889] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9776.258839] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9776.263961] lmv_intent_open+0x338/0x1150 [lmv] [ 9776.266241] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9776.269969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9776.272626] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9776.276592] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9776.284665] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9776.287797] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9776.290881] ? do_raw_spin_unlock+0x75/0x190 [ 9776.292678] ? _raw_spin_unlock+0x12/0x30 [ 9776.295270] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9776.301558] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9776.305762] ll_atomic_open+0x36a/0xd70 [lustre] [ 9776.307728] lookup_open+0x356/0xb40 [ 9776.309093] do_last+0x313/0xfc0 [ 9776.315595] ? nd_jump_root+0xe5/0x160 [ 9776.316907] ? path_init+0x437/0x520 [ 9776.318841] path_openat+0xf7/0x500 [ 9776.321222] ? do_raw_spin_unlock+0x75/0x190 [ 9776.326987] do_filp_open+0x99/0x140 [ 9776.333758] ? getname_flags+0x6e/0x330 [ 9776.336767] ? __check_object_size+0xff/0x256 [ 9776.339570] ? do_raw_spin_unlock+0x75/0x190 [ 9776.342335] ? _raw_spin_unlock+0x12/0x30 [ 9776.343810] do_sys_openat2+0x2b4/0x410 [ 9776.346786] do_sys_open+0x73/0xa0 [ 9776.351452] __x64_sys_openat+0x24/0x30 [ 9776.353419] do_syscall_64+0xc1/0x440 [ 9776.356419] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9776.360509] RIP: 0033:0x7f1a63983332 [ 9776.363221] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9776.373604] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9776.377844] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9776.383311] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9776.389254] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9776.393819] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9776.396431] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9780.058032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9780.060123] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9780.068729] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9780.068808] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9780.068816] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9780.068819] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9780.068832] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9780.068836] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9780.068841] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9780.068843] RDX: 00000000000021ab RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9780.068846] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9780.068848] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9780.068850] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9780.068854] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9780.068856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9780.068859] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9780.068867] Call Trace: [ 9780.068873] [ 9780.068878] ? show_regs.cold.9+0x22/0x2f [ 9780.068888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9780.068894] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9780.068898] ? watchdog+0x40/0x40 [ 9780.068903] ? hrtimer_interrupt+0x128/0x2c0 [ 9780.068906] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9780.068914] ? apic_timer_interrupt+0xf/0x20 [ 9780.068919] [ 9780.068922] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9780.068926] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9780.068931] do_raw_spin_lock+0x177/0x190 [ 9780.068935] _raw_spin_lock+0x21/0x30 [ 9780.068940] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9780.080690] Modules linked in: lustre(O) osp(O) ofd(O) [ 9780.138009] obd_mod_free+0x46/0x140 [ptlrpc] [ 9780.138677] mdc_commit_open+0x91/0xb0 [mdc] [ 9780.138723] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9780.142153] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9780.155741] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9780.155815] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9780.155823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9780.155826] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9780.155841] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9780.155847] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9780.155851] RAX: 0000000000007785 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9780.155853] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9780.155855] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9780.155857] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9780.155860] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9780.155863] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9780.155865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9780.155868] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9780.155878] Call Trace: [ 9780.155884] [ 9780.155890] ? show_regs.cold.9+0x22/0x2f [ 9780.155899] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9780.155906] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9780.155910] ? watchdog+0x40/0x40 [ 9780.155914] ? hrtimer_interrupt+0x128/0x2c0 [ 9780.155919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9780.155927] ? apic_timer_interrupt+0xf/0x20 [ 9780.155932] [ 9780.155935] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9780.155941] do_raw_spin_lock+0x177/0x190 [ 9780.155945] _raw_spin_lock+0x21/0x30 [ 9780.155950] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9780.156544] ? do_raw_spin_unlock+0x75/0x190 [ 9780.156611] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9780.156842] ? do_wait_intr+0xf0/0xf0 [ 9780.156849] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9780.157422] mdc_close+0x53b/0xd70 [mdc] [ 9780.157464] lmv_close+0x18d/0x490 [lmv] [ 9780.157491] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9780.157608] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9780.157701] ll_md_close+0x274/0x800 [lustre] [ 9780.157791] ll_file_release+0x128/0xc50 [lustre] [ 9780.158420] __fput+0x10d/0x440 [ 9780.158429] ____fput+0x12/0x20 [ 9780.158433] task_work_run+0xc6/0x110 [ 9780.158440] exit_to_usermode_loop+0x1dd/0x1f0 [ 9780.158446] do_syscall_64+0x3d6/0x440 [ 9780.158450] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9780.158458] RIP: 0033:0x7f1a63983cd8 [ 9780.158465] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9780.158469] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9780.158474] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9780.158476] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9780.158478] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9780.158480] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9780.158482] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9780.197102] BUG: workqueue lockup - pool [ 9780.203226] after_reply+0xe2c/0x1780 [ptlrpc] [ 9780.203531] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9780.203746] ? __next_timer_interrupt+0x160/0x160 [ 9780.203757] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9780.203966] ? do_wait_intr+0xf0/0xf0 [ 9780.203976] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9780.212143] cpus=0-3 flags=0x4 nice=0 stuck for 7757s! [ 9780.212163] Showing busy workqueues and worker pools: [ 9780.212165] workqueue events: flags=0x0 [ 9780.212169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9780.212178] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9780.212872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9780.212884] pending: sec_gc_main [ptlrpc] [ 9780.218438] kthread+0x1d1/0x200 [ 9780.218458] ? set_kthread_struct+0x70/0x70 [ 9780.218464] ret_from_fork+0x1f/0x30 [ 9780.821158] [ 9780.821897] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9780.829487] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9780.843558] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9780.848699] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9780.856395] workqueue events_long: flags=0x0 [ 9780.859655] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9780.867177] pending: lu_objects_flush [obdclass] [ 9780.873994] workqueue events_unbound: flags=0x2 [ 9780.882636] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9780.885939] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9780.891716] workqueue events_power_efficient: flags=0x80 [ 9780.894155] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9780.898547] pending: neigh_periodic_work, do_cache_clean [ 9780.903647] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9780.909752] pending: neigh_periodic_work, check_lifetime [ 9780.914839] workqueue mm_percpu_wq: flags=0x8 [ 9780.918260] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9780.927389] pending: vmstat_update [ 9780.928863] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9780.932659] pending: vmstat_update [ 9780.935380] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9780.940644] pending: vmstat_update [ 9780.942794] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9780.947173] pending: vmstat_update [ 9780.949675] workqueue writeback: flags=0x4a [ 9780.952777] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9780.959568] pending: wb_workfn, wb_workfn [ 9780.962072] workqueue xprtiod: flags=0xa [ 9780.964537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9780.968296] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9780.972477] workqueue ipv6_addrconf: flags=0x40008 [ 9780.984858] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9781.006530] pending: addrconf_verify_work [ 9781.012693] workqueue knbd0-recv: flags=0x1a [ 9781.016329] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9781.020589] in-flight: 647:recv_work [ 9781.022430] workqueue ptlrpc_pinger: flags=0x2 [ 9781.027520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9781.034265] pending: ptlrpc_pinger_main [ptlrpc] [ 9781.047778] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9771s workers=2 idle: 652 [ 9788.061864] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9788.074235] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9788.128816] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9788.134498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9788.139520] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9788.142158] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9788.160402] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9788.165280] RAX: 0000000000006f84 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9788.171493] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9788.181407] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9788.194554] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9788.207829] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9788.218807] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9788.231095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9788.237851] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9788.247167] Call Trace: [ 9788.251410] [ 9788.253833] ? show_regs.cold.9+0x22/0x2f [ 9788.254946] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9788.256274] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9788.260805] ? watchdog+0x40/0x40 [ 9788.263199] ? hrtimer_interrupt+0x128/0x2c0 [ 9788.274462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9788.281461] ? apic_timer_interrupt+0xf/0x20 [ 9788.292624] [ 9788.296508] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9788.305483] do_raw_spin_lock+0x177/0x190 [ 9788.309049] _raw_spin_lock+0x21/0x30 [ 9788.318205] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9788.323429] ? do_raw_spin_unlock+0x75/0x190 [ 9788.327948] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9788.331528] ? do_wait_intr+0xf0/0xf0 [ 9788.333586] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9788.337794] mdc_close+0x53b/0xd70 [mdc] [ 9788.340061] lmv_close+0x18d/0x490 [lmv] [ 9788.349779] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9788.363800] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9788.369068] ll_md_close+0x274/0x800 [lustre] [ 9788.378203] ll_file_release+0x128/0xc50 [lustre] [ 9788.385542] __fput+0x10d/0x440 [ 9788.389763] ____fput+0x12/0x20 [ 9788.394302] task_work_run+0xc6/0x110 [ 9788.396594] exit_to_usermode_loop+0x1dd/0x1f0 [ 9788.406147] do_syscall_64+0x3d6/0x440 [ 9788.409880] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9788.419123] RIP: 0033:0x7f1a63983cd8 [ 9788.436465] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9788.478058] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9788.490508] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9788.502161] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9788.513881] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9788.526698] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9788.536961] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9808.055123] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [ 9808.057785] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9808.058011] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9808.058047] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9808.058123] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9808.058129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9808.058131] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9808.058146] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9808.058151] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9808.058155] RAX: 0000000000005788 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9808.058158] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9808.058160] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9808.058163] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9808.058166] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9808.058170] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9808.058173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9808.058175] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9808.058184] Call Trace: [ 9808.058190] [ 9808.058195] ? show_regs.cold.9+0x22/0x2f [ 9808.058205] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9808.058212] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9808.058216] ? watchdog+0x40/0x40 [ 9808.058221] ? hrtimer_interrupt+0x128/0x2c0 [ 9808.058225] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9808.058234] ? apic_timer_interrupt+0xf/0x20 [ 9808.058238] [ 9808.058241] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9808.058246] do_raw_spin_lock+0x177/0x190 [ 9808.058251] _raw_spin_lock+0x21/0x30 [ 9808.058256] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9808.058952] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9808.059028] ? do_raw_spin_unlock+0x75/0x190 [ 9808.059039] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9808.059559] ? do_wait_intr+0xf0/0xf0 [ 9808.059569] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9808.060118] mdc_close+0x53b/0xd70 [mdc] [ 9808.060156] lmv_close+0x18d/0x490 [lmv] [ 9808.060621] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9808.061157] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9808.061237] ll_md_close+0x274/0x800 [lustre] [ 9808.061321] ll_file_release+0x128/0xc50 [lustre] [ 9808.061620] __fput+0x10d/0x440 [ 9808.061635] ____fput+0x12/0x20 [ 9808.061640] task_work_run+0xc6/0x110 [ 9808.061648] exit_to_usermode_loop+0x1dd/0x1f0 [ 9808.061656] do_syscall_64+0x3d6/0x440 [ 9808.061661] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9808.061670] RIP: 0033:0x7f1a63983cd8 [ 9808.061678] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9808.061682] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9808.061760] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9808.061764] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9808.061766] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9808.061768] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9808.061770] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9808.062337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [ 9808.065535] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9808.098026] nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9808.098076] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9808.098082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9808.098085] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9808.098098] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9808.098102] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9808.098106] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9808.098112] RDX: 00000000000020b4 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9808.098114] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9808.098116] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9808.098118] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9808.098121] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9808.098123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9808.098125] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9808.098133] Call Trace: [ 9808.098138] [ 9808.098143] ? show_regs.cold.9+0x22/0x2f [ 9808.098159] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9808.098166] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9808.098170] ? watchdog+0x40/0x40 [ 9808.098174] ? hrtimer_interrupt+0x128/0x2c0 [ 9808.098177] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9808.098186] ? apic_timer_interrupt+0xf/0x20 [ 9808.098190] [ 9808.098193] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9808.098197] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9808.098205] do_raw_spin_lock+0x177/0x190 [ 9808.098212] _raw_spin_lock+0x21/0x30 [ 9808.098218] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9808.098537] obd_mod_free+0x46/0x140 [ptlrpc] [ 9808.098777] mdc_commit_open+0x91/0xb0 [mdc] [ 9808.098837] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9808.099076] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9808.099323] after_reply+0xe2c/0x1780 [ptlrpc] [ 9808.099554] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9808.099816] ? __next_timer_interrupt+0x160/0x160 [ 9808.099827] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9808.100067] ? do_wait_intr+0xf0/0xf0 [ 9808.100077] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9808.100314] kthread+0x1d1/0x200 [ 9808.100320] ? set_kthread_struct+0x70/0x70 [ 9808.100324] ret_from_fork+0x1f/0x30 [ 9808.452671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9808.455967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9808.458299] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9808.465773] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9808.470394] RAX: 000000000000418b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9808.473920] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9808.478343] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9808.481267] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9808.484182] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9808.487301] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9808.490507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9808.492967] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9808.495671] Call Trace: [ 9808.496906] [ 9808.497896] ? show_regs.cold.9+0x22/0x2f [ 9808.499209] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9808.501372] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9808.503268] ? watchdog+0x40/0x40 [ 9808.504537] ? hrtimer_interrupt+0x128/0x2c0 [ 9808.506241] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9808.508168] ? apic_timer_interrupt+0xf/0x20 [ 9808.509965] [ 9808.510778] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9808.516480] do_raw_spin_lock+0x177/0x190 [ 9808.519155] _raw_spin_lock+0x21/0x30 [ 9808.523026] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9808.525483] ? do_raw_spin_unlock+0x75/0x190 [ 9808.528976] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9808.531488] ? do_wait_intr+0xf0/0xf0 [ 9808.533506] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9808.536230] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9808.539083] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9808.541836] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9808.544116] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9808.546671] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9808.548961] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9808.550948] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9808.553163] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9808.556225] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9808.558752] lmv_intent_open+0x338/0x1150 [lmv] [ 9808.560886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9808.563187] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9808.566362] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9808.570593] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9808.573129] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9808.574928] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9808.577325] ? do_raw_spin_unlock+0x75/0x190 [ 9808.582433] ? _raw_spin_unlock+0x12/0x30 [ 9808.590722] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9808.594160] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9808.598977] ll_atomic_open+0x36a/0xd70 [lustre] [ 9808.603237] lookup_open+0x356/0xb40 [ 9808.611443] do_last+0x313/0xfc0 [ 9808.617091] ? nd_jump_root+0xe5/0x160 [ 9808.621991] ? path_init+0x437/0x520 [ 9808.632211] path_openat+0xf7/0x500 [ 9808.635964] ? do_raw_spin_unlock+0x75/0x190 [ 9808.639047] do_filp_open+0x99/0x140 [ 9808.641087] ? getname_flags+0x6e/0x330 [ 9808.643332] ? __check_object_size+0xff/0x256 [ 9808.646144] ? do_raw_spin_unlock+0x75/0x190 [ 9808.649151] ? _raw_spin_unlock+0x12/0x30 [ 9808.654449] do_sys_openat2+0x2b4/0x410 [ 9808.658247] do_sys_open+0x73/0xa0 [ 9808.664965] __x64_sys_openat+0x24/0x30 [ 9808.672956] do_syscall_64+0xc1/0x440 [ 9808.675312] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9808.682047] RIP: 0033:0x7f1a63983332 [ 9808.684194] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9808.698194] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9808.702204] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9808.708293] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9808.712239] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9808.715581] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9808.726429] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9811.424046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7788s! [ 9811.426260] Showing busy workqueues and worker pools: [ 9811.427524] workqueue events: flags=0x0 [ 9811.429849] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9811.433177] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9811.436628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9811.439900] pending: sec_gc_main [ptlrpc] [ 9811.442421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9811.445418] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9811.450080] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9811.453147] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9811.457673] workqueue events_long: flags=0x0 [ 9811.459631] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9811.463165] pending: lu_objects_flush [obdclass] [ 9811.465702] workqueue events_unbound: flags=0x2 [ 9811.467878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9811.470916] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9811.475716] workqueue events_power_efficient: flags=0x80 [ 9811.478655] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9811.482490] pending: neigh_periodic_work, do_cache_clean [ 9811.485195] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9811.488126] pending: neigh_periodic_work, check_lifetime [ 9811.491035] workqueue mm_percpu_wq: flags=0x8 [ 9811.492690] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9811.497831] pending: vmstat_update [ 9811.504481] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9811.509155] pending: vmstat_update [ 9811.514434] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9811.521399] pending: vmstat_update [ 9811.526810] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9811.529142] pending: vmstat_update [ 9811.531242] workqueue writeback: flags=0x4a [ 9811.532963] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9811.536911] pending: wb_workfn, wb_workfn [ 9811.539405] workqueue xprtiod: flags=0xa [ 9811.541723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9811.543610] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9811.546430] workqueue ipv6_addrconf: flags=0x40008 [ 9811.548872] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9811.551255] pending: addrconf_verify_work [ 9811.552565] workqueue knbd0-recv: flags=0x1a [ 9811.553889] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9811.555225] in-flight: 647:recv_work [ 9811.556060] workqueue ptlrpc_pinger: flags=0x2 [ 9811.557868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9811.559723] pending: ptlrpc_pinger_main [ptlrpc] [ 9811.561507] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9801s workers=2 idle: 652 [ 9816.056570] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9816.060285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9816.083804] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9816.090091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9816.095276] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9816.098736] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9816.109484] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9816.113883] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9816.117903] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9816.122110] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9816.125644] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9816.129244] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9816.133077] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9816.137243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9816.139933] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9816.143475] Call Trace: [ 9816.145066] [ 9816.146090] ? show_regs.cold.9+0x22/0x2f [ 9816.147955] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9816.150236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9816.153334] ? watchdog+0x40/0x40 [ 9816.155258] ? hrtimer_interrupt+0x128/0x2c0 [ 9816.157478] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9816.160082] ? apic_timer_interrupt+0xf/0x20 [ 9816.162131] [ 9816.163365] ? native_safe_halt+0x2e/0x50 [ 9816.166055] kvm_wait+0xcd/0xe0 [ 9816.167431] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9816.170662] do_raw_spin_lock+0x177/0x190 [ 9816.173679] _raw_spin_lock+0x21/0x30 [ 9816.175894] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9816.179466] ? do_raw_spin_unlock+0x75/0x190 [ 9816.182045] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9816.185052] ? do_wait_intr+0xf0/0xf0 [ 9816.187199] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9816.190416] mdc_close+0x53b/0xd70 [mdc] [ 9816.193432] lmv_close+0x18d/0x490 [lmv] [ 9816.198149] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9816.201943] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9816.205016] ll_md_close+0x274/0x800 [lustre] [ 9816.208541] ll_file_release+0x128/0xc50 [lustre] [ 9816.212891] __fput+0x10d/0x440 [ 9816.214940] ____fput+0x12/0x20 [ 9816.217397] task_work_run+0xc6/0x110 [ 9816.219705] exit_to_usermode_loop+0x1dd/0x1f0 [ 9816.223253] do_syscall_64+0x3d6/0x440 [ 9816.225712] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9816.228191] RIP: 0033:0x7f1a63983cd8 [ 9816.230143] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9816.239889] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9816.243880] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9816.247449] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9816.251925] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9816.256053] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9816.260577] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9822.616561] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 9822.619808] rcu: 3-...!: (4923762 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [ 9822.632999] (detected by 1, t=7800210 jiffies, g=269633, q=2756) [ 9822.639853] Sending NMI from CPU 1 to CPUs 3: [ 9822.650331] NMI backtrace for cpu 3 [ 9822.650338] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9822.650341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9822.650343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9822.650344] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9822.650347] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [ 9822.650350] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9822.650351] RDX: 0000000000005fce RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9822.650352] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9822.650353] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9822.650355] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9822.650356] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9822.650358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9822.650359] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9822.650360] Call Trace: [ 9822.650361] [ 9822.650362] ? show_regs.cold.9+0x22/0x2f [ 9822.650363] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9822.650364] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9822.650365] ? nmi_handle+0xbd/0x210 [ 9822.650366] ? default_do_nmi+0x4b/0x1b0 [ 9822.650367] ? do_nmi+0x280/0x380 [ 9822.650368] ? end_repeat_nmi+0x16/0x1f [ 9822.650369] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9822.650370] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9822.650371] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9822.650372] [ 9822.650373] do_raw_spin_lock+0x177/0x190 [ 9822.650374] _raw_spin_lock+0x21/0x30 [ 9822.650375] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9822.650376] obd_mod_free+0x46/0x140 [ptlrpc] [ 9822.650377] mdc_commit_open+0x91/0xb0 [mdc] [ 9822.650378] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9822.650379] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9822.650380] after_reply+0xe2c/0x1780 [ptlrpc] [ 9822.650381] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9822.650382] ? __next_timer_interrupt+0x160/0x160 [ 9822.650383] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9822.650384] ? do_wait_intr+0xf0/0xf0 [ 9822.650385] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9822.650386] kthread+0x1d1/0x200 [ 9822.650387] ? set_kthread_struct+0x70/0x70 [ 9822.650388] ret_from_fork+0x1f/0x30 [ 9822.655385] rcu: rcu_sched kthread starved for 7799856 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 9822.945551] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9822.958096] rcu: RCU grace-period kthread stack dump: [ 9822.963223] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9822.978388] Call Trace: [ 9822.984432] __schedule+0x351/0xcb0 [ 9822.987923] schedule+0xc0/0x180 [ 9822.994478] schedule_timeout+0xb4/0x190 [ 9823.000212] ? do_raw_spin_unlock+0x75/0x190 [ 9823.014912] ? __next_timer_interrupt+0x160/0x160 [ 9823.020143] ? prepare_to_swait_event+0xf3/0x140 [ 9823.025346] rcu_gp_kthread+0x8dd/0x15b0 [ 9823.028596] ? force_qs_rnp+0x2c0/0x2c0 [ 9823.031204] kthread+0x1d1/0x200 [ 9823.035581] ? set_kthread_struct+0x70/0x70 [ 9823.038149] ret_from_fork+0x1f/0x30 [ 9823.044637] rcu: Stack dump where RCU GP kthread last ran: [ 9823.054013] Sending NMI from CPU 1 to CPUs 0: [ 9823.069709] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 11.848 msecs [ 9823.072117] NMI backtrace for cpu 0 [ 9823.072125] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9823.072126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9823.072128] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9823.072130] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9823.072132] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 [ 9823.072134] RAX: 0000000000005513 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9823.072136] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9823.072137] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9823.072139] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9823.072140] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9823.072142] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9823.072143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9823.072145] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9823.072146] Call Trace: [ 9823.072147] [ 9823.072148] ? show_regs.cold.9+0x22/0x2f [ 9823.072149] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9823.072150] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9823.072151] ? nmi_handle+0xbd/0x210 [ 9823.072152] ? default_do_nmi+0x4b/0x1b0 [ 9823.072153] ? do_nmi+0x280/0x380 [ 9823.072154] ? end_repeat_nmi+0x16/0x1f [ 9823.072156] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9823.072156] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9823.072158] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9823.072159] [ 9823.072160] do_raw_spin_lock+0x177/0x190 [ 9823.072161] _raw_spin_lock+0x21/0x30 [ 9823.072162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9823.072163] ? do_raw_spin_unlock+0x75/0x190 [ 9823.072164] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9823.072165] ? do_wait_intr+0xf0/0xf0 [ 9823.072166] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9823.072167] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9823.072168] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9823.072170] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9823.072171] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9823.072172] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9823.072173] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9823.072174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9823.072176] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9823.072177] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9823.072178] lmv_intent_open+0x338/0x1150 [lmv] [ 9823.072179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9823.072180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9823.072182] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9823.072183] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9823.072184] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9823.072186] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9823.072187] ? do_raw_spin_unlock+0x75/0x190 [ 9823.072188] ? _raw_spin_unlock+0x12/0x30 [ 9823.072189] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9823.072190] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9823.072191] ll_atomic_open+0x36a/0xd70 [lustre] [ 9823.072192] lookup_open+0x356/0xb40 [ 9823.072193] do_last+0x313/0xfc0 [ 9823.072194] ? nd_jump_root+0xe5/0x160 [ 9823.072196] ? path_init+0x437/0x520 [ 9823.072196] path_openat+0xf7/0x500 [ 9823.072197] ? do_raw_spin_unlock+0x75/0x190 [ 9823.072198] do_filp_open+0x99/0x140 [ 9823.072199] ? getname_flags+0x6e/0x330 [ 9823.072200] ? __check_object_size+0xff/0x256 [ 9823.072201] ? do_raw_spin_unlock+0x75/0x190 [ 9823.072203] ? _raw_spin_unlock+0x12/0x30 [ 9823.072204] do_sys_openat2+0x2b4/0x410 [ 9823.072206] do_sys_open+0x73/0xa0 [ 9823.072207] __x64_sys_openat+0x24/0x30 [ 9823.072208] do_syscall_64+0xc1/0x440 [ 9823.072210] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9823.072211] RIP: 0033:0x7f1a63983332 [ 9823.072213] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9823.072215] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9823.072218] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9823.072219] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9823.072220] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9823.072222] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9823.072223] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9836.057023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9836.060342] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9836.118574] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9836.127973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9836.147751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9836.150843] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9836.160897] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9836.165584] RAX: 0000000000005e04 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9836.176575] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9836.182098] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9836.185325] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9836.189798] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9836.195452] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9836.201183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9836.218892] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9836.226305] Call Trace: [ 9836.228407] [ 9836.229348] ? show_regs.cold.9+0x22/0x2f [ 9836.233344] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9836.235939] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9836.238076] ? watchdog+0x40/0x40 [ 9836.239978] ? hrtimer_interrupt+0x128/0x2c0 [ 9836.242321] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9836.244355] ? apic_timer_interrupt+0xf/0x20 [ 9836.246113] [ 9836.246869] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9836.249252] do_raw_spin_lock+0x177/0x190 [ 9836.251381] _raw_spin_lock+0x21/0x30 [ 9836.253191] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9836.255804] ? do_raw_spin_unlock+0x75/0x190 [ 9836.257815] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9836.260379] ? do_wait_intr+0xf0/0xf0 [ 9836.262356] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9836.264962] mdc_close+0x53b/0xd70 [mdc] [ 9836.266883] lmv_close+0x18d/0x490 [lmv] [ 9836.269399] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9836.272225] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9836.274303] ll_md_close+0x274/0x800 [lustre] [ 9836.275669] ll_file_release+0x128/0xc50 [lustre] [ 9836.277921] __fput+0x10d/0x440 [ 9836.279285] ____fput+0x12/0x20 [ 9836.281221] task_work_run+0xc6/0x110 [ 9836.283252] exit_to_usermode_loop+0x1dd/0x1f0 [ 9836.285287] do_syscall_64+0x3d6/0x440 [ 9836.286965] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9836.289144] RIP: 0033:0x7f1a63983cd8 [ 9836.290894] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9836.305607] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9836.310976] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9836.316609] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9836.318165] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9836.323377] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9836.327297] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9841.633208] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7818s! [ 9841.635708] Showing busy workqueues and worker pools: [ 9841.657485] workqueue events: flags=0x0 [ 9841.665685] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9841.681398] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9841.701461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9841.716941] pending: sec_gc_main [ptlrpc] [ 9841.749158] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9841.759936] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9841.779886] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9841.797199] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9841.810025] workqueue events_long: flags=0x0 [ 9841.823321] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9841.834837] pending: lu_objects_flush [obdclass] [ 9841.847108] workqueue events_unbound: flags=0x2 [ 9841.855421] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9841.873978] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9841.895342] workqueue events_power_efficient: flags=0x80 [ 9841.899275] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9841.906181] pending: neigh_periodic_work, do_cache_clean [ 9841.914370] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9841.922269] pending: neigh_periodic_work, check_lifetime [ 9841.928564] workqueue mm_percpu_wq: flags=0x8 [ 9841.934960] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9841.940782] pending: vmstat_update [ 9841.942084] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9841.957988] pending: vmstat_update [ 9841.960255] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9841.966232] pending: vmstat_update [ 9841.973014] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9841.983072] pending: vmstat_update [ 9841.984233] workqueue writeback: flags=0x4a [ 9841.985430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9842.006258] pending: wb_workfn, wb_workfn [ 9842.016894] workqueue xprtiod: flags=0xa [ 9842.024286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9842.036288] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9842.049130] workqueue ipv6_addrconf: flags=0x40008 [ 9842.050977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9842.063926] pending: addrconf_verify_work [ 9842.067094] workqueue knbd0-recv: flags=0x1a [ 9842.069420] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9842.080019] in-flight: 647:recv_work [ 9842.082924] workqueue ptlrpc_pinger: flags=0x2 [ 9842.088203] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9842.092979] pending: ptlrpc_pinger_main [ptlrpc] [ 9842.099180] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9832s workers=2 idle: 652 [ 9848.059170] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9848.062387] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9848.066535] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9848.066623] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9848.066630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9848.066633] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9848.066645] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9848.066650] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9848.066654] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9848.066657] RDX: 0000000000003b48 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9848.066659] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9848.066661] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9848.066663] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9848.066667] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9848.066674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9848.066679] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9848.066688] Call Trace: [ 9848.066693] [ 9848.066699] ? show_regs.cold.9+0x22/0x2f [ 9848.066709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9848.066716] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9848.066720] ? watchdog+0x40/0x40 [ 9848.066725] ? hrtimer_interrupt+0x128/0x2c0 [ 9848.066729] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9848.066738] ? apic_timer_interrupt+0xf/0x20 [ 9848.066747] [ 9848.066750] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9848.066755] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9848.066763] do_raw_spin_lock+0x177/0x190 [ 9848.066768] _raw_spin_lock+0x21/0x30 [ 9848.066778] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9848.067353] obd_mod_free+0x46/0x140 [ptlrpc] [ 9848.067672] mdc_commit_open+0x91/0xb0 [mdc] [ 9848.067711] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9848.067912] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9848.068257] after_reply+0xe2c/0x1780 [ptlrpc] [ 9848.068613] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9848.068862] ? __next_timer_interrupt+0x160/0x160 [ 9848.068876] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9848.069993] ? do_wait_intr+0xf0/0xf0 [ 9848.081194] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9848.081341] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9848.081352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9848.081355] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9848.081370] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9848.081374] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9848.081378] RAX: 0000000000006c87 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9848.081381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9848.081383] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9848.081385] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9848.081388] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9848.081391] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9848.081394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9848.081455] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9848.081464] Call Trace: [ 9848.081469] [ 9848.081475] ? show_regs.cold.9+0x22/0x2f [ 9848.081485] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9848.081491] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9848.081495] ? watchdog+0x40/0x40 [ 9848.081500] ? hrtimer_interrupt+0x128/0x2c0 [ 9848.081504] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9848.081512] ? apic_timer_interrupt+0xf/0x20 [ 9848.081517] [ 9848.081520] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9848.081533] do_raw_spin_lock+0x177/0x190 [ 9848.081537] _raw_spin_lock+0x21/0x30 [ 9848.081542] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9848.082030] ? do_raw_spin_unlock+0x75/0x190 [ 9848.082037] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9848.082697] ? do_wait_intr+0xf0/0xf0 [ 9848.132038] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9848.132557] kthread+0x1d1/0x200 [ 9848.132573] ? set_kthread_struct+0x70/0x70 [ 9848.132578] ret_from_fork+0x1f/0x30 [ 9848.639094] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9848.648376] mdc_close+0x53b/0xd70 [mdc] [ 9848.657762] lmv_close+0x18d/0x490 [lmv] [ 9848.662592] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9848.667005] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9848.672891] ll_md_close+0x274/0x800 [lustre] [ 9848.680742] ll_file_release+0x128/0xc50 [lustre] [ 9848.687740] __fput+0x10d/0x440 [ 9848.692223] ____fput+0x12/0x20 [ 9848.694798] task_work_run+0xc6/0x110 [ 9848.698193] exit_to_usermode_loop+0x1dd/0x1f0 [ 9848.700846] do_syscall_64+0x3d6/0x440 [ 9848.708123] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9848.709989] RIP: 0033:0x7f1a63983cd8 [ 9848.720510] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9848.747698] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9848.760728] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9848.767786] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9848.774552] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9848.791846] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9848.802936] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9864.057030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9864.060288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9864.102231] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9864.115074] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9864.127639] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9864.131210] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9864.149965] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9864.154684] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9864.164087] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9864.166687] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9864.176344] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9864.178988] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9864.181608] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9864.185008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9864.189189] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9864.193013] Call Trace: [ 9864.195370] [ 9864.196794] ? show_regs.cold.9+0x22/0x2f [ 9864.198475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9864.201812] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9864.203739] ? watchdog+0x40/0x40 [ 9864.205664] ? hrtimer_interrupt+0x128/0x2c0 [ 9864.207753] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9864.209814] ? apic_timer_interrupt+0xf/0x20 [ 9864.211643] [ 9864.212458] ? native_safe_halt+0x2e/0x50 [ 9864.214108] kvm_wait+0xcd/0xe0 [ 9864.215797] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9864.222267] do_raw_spin_lock+0x177/0x190 [ 9864.225391] _raw_spin_lock+0x21/0x30 [ 9864.226635] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9864.229910] ? do_raw_spin_unlock+0x75/0x190 [ 9864.231924] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9864.237106] ? do_wait_intr+0xf0/0xf0 [ 9864.239266] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9864.244633] mdc_close+0x53b/0xd70 [mdc] [ 9864.245858] lmv_close+0x18d/0x490 [lmv] [ 9864.251653] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9864.259531] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9864.261691] ll_md_close+0x274/0x800 [lustre] [ 9864.264822] ll_file_release+0x128/0xc50 [lustre] [ 9864.267870] __fput+0x10d/0x440 [ 9864.269398] ____fput+0x12/0x20 [ 9864.270647] task_work_run+0xc6/0x110 [ 9864.272198] exit_to_usermode_loop+0x1dd/0x1f0 [ 9864.274088] do_syscall_64+0x3d6/0x440 [ 9864.276171] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9864.278003] RIP: 0033:0x7f1a63983cd8 [ 9864.281999] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9864.294825] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9864.299324] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9864.302398] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9864.307867] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9864.311102] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9864.313976] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9868.053266] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 9868.057188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9868.096802] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9868.102427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9868.111457] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9868.115987] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9868.125918] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9868.129538] RAX: 000000000000750e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9868.132804] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9868.136047] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9868.139490] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9868.142395] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9868.144960] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9868.148290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9868.150584] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9868.153625] Call Trace: [ 9868.154586] [ 9868.155427] ? show_regs.cold.9+0x22/0x2f [ 9868.157145] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9868.159222] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9868.161269] ? watchdog+0x40/0x40 [ 9868.162538] ? hrtimer_interrupt+0x128/0x2c0 [ 9868.164330] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9868.166181] ? apic_timer_interrupt+0xf/0x20 [ 9868.168421] [ 9868.169120] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9868.171122] do_raw_spin_lock+0x177/0x190 [ 9868.172495] _raw_spin_lock+0x21/0x30 [ 9868.174022] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9868.176470] ? do_raw_spin_unlock+0x75/0x190 [ 9868.178405] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9868.180695] ? do_wait_intr+0xf0/0xf0 [ 9868.182370] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9868.184790] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9868.188074] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9868.191533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9868.194729] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9868.197867] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9868.200499] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9868.202821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9868.205887] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9868.209024] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9868.211758] lmv_intent_open+0x338/0x1150 [lmv] [ 9868.214277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9868.216692] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9868.219256] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9868.221308] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9868.223547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9868.225662] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9868.227751] ? do_raw_spin_unlock+0x75/0x190 [ 9868.229606] ? _raw_spin_unlock+0x12/0x30 [ 9868.231561] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9868.233803] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9868.235911] ll_atomic_open+0x36a/0xd70 [lustre] [ 9868.237896] lookup_open+0x356/0xb40 [ 9868.239387] do_last+0x313/0xfc0 [ 9868.240778] ? nd_jump_root+0xe5/0x160 [ 9868.242397] ? path_init+0x437/0x520 [ 9868.244812] path_openat+0xf7/0x500 [ 9868.246334] ? do_raw_spin_unlock+0x75/0x190 [ 9868.248566] do_filp_open+0x99/0x140 [ 9868.256764] ? getname_flags+0x6e/0x330 [ 9868.261455] ? __check_object_size+0xff/0x256 [ 9868.263004] ? do_raw_spin_unlock+0x75/0x190 [ 9868.265532] ? _raw_spin_unlock+0x12/0x30 [ 9868.268067] do_sys_openat2+0x2b4/0x410 [ 9868.270806] do_sys_open+0x73/0xa0 [ 9868.272607] __x64_sys_openat+0x24/0x30 [ 9868.274805] do_syscall_64+0xc1/0x440 [ 9868.276235] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9868.287313] RIP: 0033:0x7f1a63983332 [ 9868.288708] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9868.305498] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9868.309072] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9868.314620] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9868.325833] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9868.330769] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9868.340931] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9872.355084] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7849s! [ 9872.359062] Showing busy workqueues and worker pools: [ 9872.361920] workqueue events: flags=0x0 [ 9872.364262] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9872.368190] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9872.373839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9872.380043] pending: sec_gc_main [ptlrpc] [ 9872.383408] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9872.387583] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9872.392737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9872.396146] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9872.401131] workqueue events_long: flags=0x0 [ 9872.402925] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9872.407259] pending: lu_objects_flush [obdclass] [ 9872.411092] workqueue events_unbound: flags=0x2 [ 9872.413867] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9872.418186] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9872.425046] workqueue events_power_efficient: flags=0x80 [ 9872.427868] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9872.431016] pending: neigh_periodic_work, do_cache_clean [ 9872.433313] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9872.437284] pending: neigh_periodic_work, check_lifetime [ 9872.441136] workqueue mm_percpu_wq: flags=0x8 [ 9872.443976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9872.447816] pending: vmstat_update [ 9872.449907] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9872.454652] pending: vmstat_update [ 9872.457034] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9872.459979] pending: vmstat_update [ 9872.461886] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9872.464914] pending: vmstat_update [ 9872.466723] workqueue writeback: flags=0x4a [ 9872.469268] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9872.472816] pending: wb_workfn, wb_workfn [ 9872.474916] workqueue xprtiod: flags=0xa [ 9872.476944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9872.480157] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9872.484868] workqueue ipv6_addrconf: flags=0x40008 [ 9872.486602] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9872.489797] pending: addrconf_verify_work [ 9872.491811] workqueue knbd0-recv: flags=0x1a [ 9872.493901] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9872.497603] in-flight: 647:recv_work [ 9872.499699] workqueue ptlrpc_pinger: flags=0x2 [ 9872.501902] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9872.504941] pending: ptlrpc_pinger_main [ptlrpc] [ 9872.507723] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9862s workers=2 idle: 652 [ 9876.056083] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9876.058807] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9876.059763] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9876.059835] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9876.059842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9876.059845] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9876.059858] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9876.059862] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9876.059867] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9876.059869] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9876.059872] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9876.059874] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9876.059876] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9876.059879] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9876.059882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9876.059885] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9876.059894] Call Trace: [ 9876.059900] [ 9876.059907] ? show_regs.cold.9+0x22/0x2f [ 9876.059918] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9876.059924] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9876.059929] ? watchdog+0x40/0x40 [ 9876.059932] ? hrtimer_interrupt+0x128/0x2c0 [ 9876.059936] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9876.059943] ? apic_timer_interrupt+0xf/0x20 [ 9876.059947] [ 9876.059950] ? native_safe_halt+0x2e/0x50 [ 9876.059954] kvm_wait+0xcd/0xe0 [ 9876.059961] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9876.059969] do_raw_spin_lock+0x177/0x190 [ 9876.059973] _raw_spin_lock+0x21/0x30 [ 9876.059977] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9876.060341] ? do_raw_spin_unlock+0x75/0x190 [ 9876.065610] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9876.065702] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9876.065710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9876.066043] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9876.088468] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9876.088746] ? do_wait_intr+0xf0/0xf0 [ 9876.088756] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9876.088951] mdc_close+0x53b/0xd70 [mdc] [ 9876.088982] lmv_close+0x18d/0x490 [lmv] [ 9876.098121] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9876.098340] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9876.098347] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9876.098349] RDX: 0000000000000cd6 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9876.098352] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9876.098354] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9876.098356] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9876.098421] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9876.098424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9876.098426] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9876.098434] Call Trace: [ 9876.098441] [ 9876.098447] ? show_regs.cold.9+0x22/0x2f [ 9876.098459] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9876.098466] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9876.098469] ? watchdog+0x40/0x40 [ 9876.098474] ? hrtimer_interrupt+0x128/0x2c0 [ 9876.098478] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9876.098486] ? apic_timer_interrupt+0xf/0x20 [ 9876.098491] [ 9876.098494] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9876.098500] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9876.098505] do_raw_spin_lock+0x177/0x190 [ 9876.098510] _raw_spin_lock+0x21/0x30 [ 9876.098517] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9876.098807] obd_mod_free+0x46/0x140 [ptlrpc] [ 9876.099000] mdc_commit_open+0x91/0xb0 [mdc] [ 9876.099036] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9876.102675] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9876.102871] after_reply+0xe2c/0x1780 [ptlrpc] [ 9876.103065] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9876.119891] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9876.120010] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9876.120092] ll_md_close+0x274/0x800 [lustre] [ 9876.120285] ll_file_release+0x128/0xc50 [lustre] [ 9876.120421] __fput+0x10d/0x440 [ 9876.120430] ____fput+0x12/0x20 [ 9876.120434] task_work_run+0xc6/0x110 [ 9876.120442] exit_to_usermode_loop+0x1dd/0x1f0 [ 9876.120449] do_syscall_64+0x3d6/0x440 [ 9876.120453] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9876.120462] RIP: 0033:0x7f1a63983cd8 [ 9876.120469] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9876.120643] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9876.120651] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9876.120653] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9876.120655] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9876.120657] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9876.120659] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9876.597752] ? __next_timer_interrupt+0x160/0x160 [ 9876.601017] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9876.608423] ? do_wait_intr+0xf0/0xf0 [ 9876.612105] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9876.617453] kthread+0x1d1/0x200 [ 9876.623524] ? set_kthread_struct+0x70/0x70 [ 9876.630720] ret_from_fork+0x1f/0x30 [ 9892.057108] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9892.065420] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9892.099884] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9892.108061] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9892.113480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9892.120478] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9892.134518] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9892.138491] RAX: 0000000000005b8f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9892.141925] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9892.145962] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9892.149935] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9892.154385] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9892.157824] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9892.162545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9892.165567] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9892.169050] Call Trace: [ 9892.170281] [ 9892.171063] ? show_regs.cold.9+0x22/0x2f [ 9892.172578] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9892.174960] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9892.177050] ? watchdog+0x40/0x40 [ 9892.179273] ? hrtimer_interrupt+0x128/0x2c0 [ 9892.181429] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9892.184430] ? apic_timer_interrupt+0xf/0x20 [ 9892.187062] [ 9892.188122] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9892.190918] do_raw_spin_lock+0x177/0x190 [ 9892.192705] _raw_spin_lock+0x21/0x30 [ 9892.194652] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9892.197828] ? do_raw_spin_unlock+0x75/0x190 [ 9892.199687] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9892.202363] ? do_wait_intr+0xf0/0xf0 [ 9892.204078] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9892.207165] mdc_close+0x53b/0xd70 [mdc] [ 9892.209385] lmv_close+0x18d/0x490 [lmv] [ 9892.211357] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9892.213742] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9892.216323] ll_md_close+0x274/0x800 [lustre] [ 9892.218512] ll_file_release+0x128/0xc50 [lustre] [ 9892.221174] __fput+0x10d/0x440 [ 9892.223030] ____fput+0x12/0x20 [ 9892.224447] task_work_run+0xc6/0x110 [ 9892.225968] exit_to_usermode_loop+0x1dd/0x1f0 [ 9892.228553] do_syscall_64+0x3d6/0x440 [ 9892.230931] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9892.233624] RIP: 0033:0x7f1a63983cd8 [ 9892.235609] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9892.244674] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9892.248525] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9892.252256] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9892.255603] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9892.263457] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9892.268623] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9900.053012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 9900.056939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9900.083978] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9900.090785] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9900.095219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9900.103289] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9900.119982] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9900.127349] RAX: 0000000000004cd8 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9900.131684] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9900.134798] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9900.140147] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9900.143514] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9900.151546] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9900.157917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9900.161290] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9900.165280] Call Trace: [ 9900.166468] [ 9900.167443] ? show_regs.cold.9+0x22/0x2f [ 9900.169571] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9900.173307] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9900.178625] ? watchdog+0x40/0x40 [ 9900.181378] ? hrtimer_interrupt+0x128/0x2c0 [ 9900.183992] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9900.191148] ? apic_timer_interrupt+0xf/0x20 [ 9900.193460] [ 9900.195062] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9900.197562] do_raw_spin_lock+0x177/0x190 [ 9900.200734] _raw_spin_lock+0x21/0x30 [ 9900.202666] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9900.205399] ? do_raw_spin_unlock+0x75/0x190 [ 9900.207948] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9900.213331] ? do_wait_intr+0xf0/0xf0 [ 9900.219055] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9900.222298] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9900.225850] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9900.232300] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9900.236138] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9900.239473] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9900.242115] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9900.246342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9900.249749] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9900.253863] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9900.256566] lmv_intent_open+0x338/0x1150 [lmv] [ 9900.266885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9900.272113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9900.274640] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9900.276862] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9900.279666] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9900.282291] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9900.285510] ? do_raw_spin_unlock+0x75/0x190 [ 9900.287210] ? _raw_spin_unlock+0x12/0x30 [ 9900.288755] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9900.294628] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9900.299645] ll_atomic_open+0x36a/0xd70 [lustre] [ 9900.301755] lookup_open+0x356/0xb40 [ 9900.304380] do_last+0x313/0xfc0 [ 9900.307608] ? nd_jump_root+0xe5/0x160 [ 9900.312527] ? path_init+0x437/0x520 [ 9900.314618] path_openat+0xf7/0x500 [ 9900.316853] ? do_raw_spin_unlock+0x75/0x190 [ 9900.318668] do_filp_open+0x99/0x140 [ 9900.320287] ? getname_flags+0x6e/0x330 [ 9900.322253] ? __check_object_size+0xff/0x256 [ 9900.324476] ? do_raw_spin_unlock+0x75/0x190 [ 9900.326376] ? _raw_spin_unlock+0x12/0x30 [ 9900.330136] do_sys_openat2+0x2b4/0x410 [ 9900.331553] do_sys_open+0x73/0xa0 [ 9900.332712] __x64_sys_openat+0x24/0x30 [ 9900.342716] do_syscall_64+0xc1/0x440 [ 9900.344875] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9900.347078] RIP: 0033:0x7f1a63983332 [ 9900.348987] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9900.360074] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9900.367063] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9900.375637] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9900.380997] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9900.383655] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9900.388463] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9902.560874] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7879s! [ 9902.564060] Showing busy workqueues and worker pools: [ 9902.566145] workqueue events: flags=0x0 [ 9902.568475] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9902.571708] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9902.581206] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9902.587003] pending: sec_gc_main [ptlrpc] [ 9902.589556] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9902.592402] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9902.596289] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9902.598917] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9902.602803] workqueue events_long: flags=0x0 [ 9902.604496] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9902.606794] pending: lu_objects_flush [obdclass] [ 9902.608548] workqueue events_unbound: flags=0x2 [ 9902.610179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9902.612250] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9902.616271] workqueue events_power_efficient: flags=0x80 [ 9902.617930] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9902.620524] pending: neigh_periodic_work, do_cache_clean [ 9902.622765] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9902.625534] pending: neigh_periodic_work, check_lifetime [ 9902.627796] workqueue mm_percpu_wq: flags=0x8 [ 9902.629570] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9902.632787] pending: vmstat_update [ 9902.634434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9902.637356] pending: vmstat_update [ 9902.639160] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9902.641636] pending: vmstat_update [ 9902.643345] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9902.645279] pending: vmstat_update [ 9902.646714] workqueue writeback: flags=0x4a [ 9902.647826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9902.650013] pending: wb_workfn, wb_workfn [ 9902.651282] workqueue xprtiod: flags=0xa [ 9902.652912] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9902.654931] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9902.657968] workqueue ipv6_addrconf: flags=0x40008 [ 9902.664489] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9902.670711] pending: addrconf_verify_work [ 9902.682504] workqueue knbd0-recv: flags=0x1a [ 9902.690034] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9902.696443] in-flight: 647:recv_work [ 9902.698347] workqueue ptlrpc_pinger: flags=0x2 [ 9902.700855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9902.704141] pending: ptlrpc_pinger_main [ptlrpc] [ 9902.706829] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9892s workers=2 idle: 652 [ 9904.057354] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9904.058631] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9904.068162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9904.068247] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9904.068254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9904.068258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9904.068272] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9904.068277] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9904.068281] RAX: 0000000000006985 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9904.068284] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9904.068286] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9904.068289] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9904.068291] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9904.068294] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9904.068297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9904.068299] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9904.068308] Call Trace: [ 9904.068314] [ 9904.068319] ? show_regs.cold.9+0x22/0x2f [ 9904.068329] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9904.068336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9904.068341] ? watchdog+0x40/0x40 [ 9904.068345] ? hrtimer_interrupt+0x128/0x2c0 [ 9904.068349] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9904.068358] ? apic_timer_interrupt+0xf/0x20 [ 9904.068362] [ 9904.068365] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9904.068372] do_raw_spin_lock+0x177/0x190 [ 9904.068377] _raw_spin_lock+0x21/0x30 [ 9904.068381] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9904.071544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9904.071614] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9904.071620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9904.071623] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9904.071635] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9904.071639] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9904.071643] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9904.071645] RDX: 0000000000006743 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9904.071647] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9904.071649] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9904.071651] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9904.071654] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9904.071657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9904.071659] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9904.071665] Call Trace: [ 9904.071670] [ 9904.071674] ? show_regs.cold.9+0x22/0x2f [ 9904.071681] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9904.071686] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9904.071690] ? watchdog+0x40/0x40 [ 9904.071693] ? hrtimer_interrupt+0x128/0x2c0 [ 9904.071697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9904.071703] ? apic_timer_interrupt+0xf/0x20 [ 9904.071707] [ 9904.071709] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9904.071713] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9904.071717] do_raw_spin_lock+0x177/0x190 [ 9904.071721] _raw_spin_lock+0x21/0x30 [ 9904.071725] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9904.072021] obd_mod_free+0x46/0x140 [ptlrpc] [ 9904.072216] mdc_commit_open+0x91/0xb0 [mdc] [ 9904.072256] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9904.072764] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9904.073023] after_reply+0xe2c/0x1780 [ptlrpc] [ 9904.073229] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9904.137477] ? do_raw_spin_unlock+0x75/0x190 [ 9904.149338] ? __next_timer_interrupt+0x160/0x160 [ 9904.149363] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9904.151565] ? do_wait_intr+0xf0/0xf0 [ 9904.151593] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9904.151811] kthread+0x1d1/0x200 [ 9904.151817] ? set_kthread_struct+0x70/0x70 [ 9904.151821] ret_from_fork+0x1f/0x30 [ 9904.581513] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9904.583689] ? do_wait_intr+0xf0/0xf0 [ 9904.590592] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9904.597480] mdc_close+0x53b/0xd70 [mdc] [ 9904.607607] lmv_close+0x18d/0x490 [lmv] [ 9904.618091] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9904.620022] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9904.623336] ll_md_close+0x274/0x800 [lustre] [ 9904.627448] ll_file_release+0x128/0xc50 [lustre] [ 9904.634488] __fput+0x10d/0x440 [ 9904.636596] ____fput+0x12/0x20 [ 9904.638378] task_work_run+0xc6/0x110 [ 9904.640354] exit_to_usermode_loop+0x1dd/0x1f0 [ 9904.644912] do_syscall_64+0x3d6/0x440 [ 9904.646373] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9904.649605] RIP: 0033:0x7f1a63983cd8 [ 9904.652451] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9904.663373] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9904.666588] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9904.669248] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9904.672511] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9904.677200] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9904.680431] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9920.057034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9920.061464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9920.091967] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9920.098305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9920.105069] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9920.111088] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9920.129676] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9920.135309] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9920.138413] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9920.141945] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9920.145316] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9920.148860] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9920.152393] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9920.155797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9920.157795] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9920.160595] Call Trace: [ 9920.161444] [ 9920.162278] ? show_regs.cold.9+0x22/0x2f [ 9920.163457] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9920.164986] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9920.166359] ? watchdog+0x40/0x40 [ 9920.167392] ? hrtimer_interrupt+0x128/0x2c0 [ 9920.168673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9920.170448] ? apic_timer_interrupt+0xf/0x20 [ 9920.171866] [ 9920.172495] ? native_safe_halt+0x2e/0x50 [ 9920.173882] kvm_wait+0xcd/0xe0 [ 9920.174944] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9920.176456] do_raw_spin_lock+0x177/0x190 [ 9920.177828] _raw_spin_lock+0x21/0x30 [ 9920.178966] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9920.180849] ? do_raw_spin_unlock+0x75/0x190 [ 9920.182224] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9920.183994] ? do_wait_intr+0xf0/0xf0 [ 9920.185230] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9920.187189] mdc_close+0x53b/0xd70 [mdc] [ 9920.188584] lmv_close+0x18d/0x490 [lmv] [ 9920.189920] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9920.191853] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9920.193362] ll_md_close+0x274/0x800 [lustre] [ 9920.194907] ll_file_release+0x128/0xc50 [lustre] [ 9920.196382] __fput+0x10d/0x440 [ 9920.197467] ____fput+0x12/0x20 [ 9920.198448] task_work_run+0xc6/0x110 [ 9920.199554] exit_to_usermode_loop+0x1dd/0x1f0 [ 9920.201125] do_syscall_64+0x3d6/0x440 [ 9920.202275] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9920.204104] RIP: 0033:0x7f1a63983cd8 [ 9920.205237] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9920.211274] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9920.213794] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9920.216209] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9920.218780] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9920.221693] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9920.224689] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9928.056437] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 9928.066538] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9928.122074] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9928.131978] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9928.146243] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9928.156776] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9928.178524] RSP: 0018:ffffc38541c37500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9928.185057] RAX: 0000000000007f8d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9928.191595] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9928.201329] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9928.203704] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9928.210499] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9928.220500] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9928.226896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9928.230848] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9928.239195] Call Trace: [ 9928.240681] [ 9928.241915] ? show_regs.cold.9+0x22/0x2f [ 9928.244895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9928.250364] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9928.254939] ? watchdog+0x40/0x40 [ 9928.259908] ? hrtimer_interrupt+0x128/0x2c0 [ 9928.262136] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9928.266593] ? apic_timer_interrupt+0xf/0x20 [ 9928.268983] [ 9928.273175] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9928.279240] do_raw_spin_lock+0x177/0x190 [ 9928.287146] _raw_spin_lock+0x21/0x30 [ 9928.290370] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9928.302749] ? do_raw_spin_unlock+0x75/0x190 [ 9928.304336] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9928.310861] ? do_wait_intr+0xf0/0xf0 [ 9928.313890] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9928.319780] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9928.322334] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9928.324164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9928.329111] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9928.334984] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9928.345644] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9928.350435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9928.354143] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9928.363799] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9928.369088] lmv_intent_open+0x338/0x1150 [lmv] [ 9928.373856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9928.378535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9928.383435] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9928.386452] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9928.395584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9928.402786] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9928.406607] ? do_raw_spin_unlock+0x75/0x190 [ 9928.410199] ? _raw_spin_unlock+0x12/0x30 [ 9928.414294] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9928.416917] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9928.422522] ll_atomic_open+0x36a/0xd70 [lustre] [ 9928.426152] lookup_open+0x356/0xb40 [ 9928.429541] do_last+0x313/0xfc0 [ 9928.433285] ? nd_jump_root+0xe5/0x160 [ 9928.437488] ? path_init+0x437/0x520 [ 9928.441432] path_openat+0xf7/0x500 [ 9928.442889] ? do_raw_spin_unlock+0x75/0x190 [ 9928.447339] do_filp_open+0x99/0x140 [ 9928.448939] ? getname_flags+0x6e/0x330 [ 9928.453224] ? __check_object_size+0xff/0x256 [ 9928.459782] ? do_raw_spin_unlock+0x75/0x190 [ 9928.463634] ? _raw_spin_unlock+0x12/0x30 [ 9928.467036] do_sys_openat2+0x2b4/0x410 [ 9928.468928] do_sys_open+0x73/0xa0 [ 9928.470460] __x64_sys_openat+0x24/0x30 [ 9928.476584] do_syscall_64+0xc1/0x440 [ 9928.478435] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9928.481464] RIP: 0033:0x7f1a63983332 [ 9928.483579] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9928.507092] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9928.516437] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9928.521508] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9928.527002] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9928.531762] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9928.539756] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9932.056008] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [ 9932.060479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9932.065582] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [ 9932.083154] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9932.083172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9932.083175] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [ 9932.083189] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [ 9932.083193] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9932.083199] RAX: 0000000000000c2b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9932.083201] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9932.083203] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9932.083209] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9932.083212] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9932.083215] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9932.083218] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9932.083220] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9932.083227] Call Trace: [ 9932.083233] [ 9932.083238] ? show_regs.cold.9+0x22/0x2f [ 9932.083247] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9932.083254] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9932.083258] ? watchdog+0x40/0x40 [ 9932.083269] ? hrtimer_interrupt+0x128/0x2c0 [ 9932.083273] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9932.083282] ? apic_timer_interrupt+0xf/0x20 [ 9932.083291] [ 9932.083293] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [ 9932.083300] do_raw_spin_lock+0x177/0x190 [ 9932.083309] _raw_spin_lock+0x21/0x30 [ 9932.083314] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9932.083806] ? do_raw_spin_unlock+0x75/0x190 [ 9932.083819] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9932.084029] ? do_wait_intr+0xf0/0xf0 [ 9932.084037] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9932.084333] mdc_close+0x53b/0xd70 [mdc] [ 9932.084441] lmv_close+0x18d/0x490 [lmv] [ 9932.084474] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9932.093185] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9932.093309] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9932.093316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9932.093318] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9932.093339] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9932.093343] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9932.093347] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9932.093350] RDX: 000000000000754f RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9932.093355] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9932.093358] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9932.093362] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9932.093366] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9932.093369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9932.093371] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9932.093379] Call Trace: [ 9932.093452] [ 9932.093457] ? show_regs.cold.9+0x22/0x2f [ 9932.093466] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9932.093477] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9932.093481] ? watchdog+0x40/0x40 [ 9932.093485] ? hrtimer_interrupt+0x128/0x2c0 [ 9932.093489] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9932.093496] ? apic_timer_interrupt+0xf/0x20 [ 9932.093501] [ 9932.093509] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9932.093514] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9932.093518] do_raw_spin_lock+0x177/0x190 [ 9932.093524] _raw_spin_lock+0x21/0x30 [ 9932.093533] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9932.094053] obd_mod_free+0x46/0x140 [ptlrpc] [ 9932.094283] mdc_commit_open+0x91/0xb0 [mdc] [ 9932.094344] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9932.094907] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9932.095106] after_reply+0xe2c/0x1780 [ptlrpc] [ 9932.095796] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9932.101890] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9932.102026] ll_md_close+0x274/0x800 [lustre] [ 9932.102113] ll_file_release+0x128/0xc50 [lustre] [ 9932.102211] __fput+0x10d/0x440 [ 9932.102222] ____fput+0x12/0x20 [ 9932.102227] task_work_run+0xc6/0x110 [ 9932.102234] exit_to_usermode_loop+0x1dd/0x1f0 [ 9932.102249] do_syscall_64+0x3d6/0x440 [ 9932.102254] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9932.102262] RIP: 0033:0x7f1a63983cd8 [ 9932.102269] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9932.102273] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9932.102282] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9932.102284] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9932.102286] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9932.102289] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9932.102291] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9932.380854] ? __next_timer_interrupt+0x160/0x160 [ 9932.382874] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9932.385989] ? do_wait_intr+0xf0/0xf0 [ 9932.387692] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9932.390384] kthread+0x1d1/0x200 [ 9932.391977] ? set_kthread_struct+0x70/0x70 [ 9932.393867] ret_from_fork+0x1f/0x30 [ 9932.769187] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7909s! [ 9932.771954] Showing busy workqueues and worker pools: [ 9932.773360] workqueue events: flags=0x0 [ 9932.787847] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9932.795164] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9932.808118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9932.819168] pending: sec_gc_main [ptlrpc] [ 9932.825274] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9932.829422] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9932.840343] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9932.849835] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9932.856882] workqueue events_long: flags=0x0 [ 9932.858049] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9932.869319] pending: lu_objects_flush [obdclass] [ 9932.875091] workqueue events_unbound: flags=0x2 [ 9932.878251] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9932.885498] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9932.895368] workqueue events_power_efficient: flags=0x80 [ 9932.901145] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9932.906475] pending: neigh_periodic_work, do_cache_clean [ 9932.909926] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9932.925339] pending: neigh_periodic_work, check_lifetime [ 9932.928222] workqueue mm_percpu_wq: flags=0x8 [ 9932.936976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9932.944769] pending: vmstat_update [ 9932.947662] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9932.952992] pending: vmstat_update [ 9932.959021] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9932.965008] pending: vmstat_update [ 9932.969025] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9932.976848] pending: vmstat_update [ 9932.980304] workqueue writeback: flags=0x4a [ 9932.985880] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9932.992216] pending: wb_workfn, wb_workfn [ 9933.011468] workqueue xprtiod: flags=0xa [ 9933.012959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9933.019990] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9933.030989] workqueue ipv6_addrconf: flags=0x40008 [ 9933.039376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9933.051327] pending: addrconf_verify_work [ 9933.053120] workqueue knbd0-recv: flags=0x1a [ 9933.059838] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9933.065443] in-flight: 647:recv_work [ 9933.071561] workqueue ptlrpc_pinger: flags=0x2 [ 9933.077788] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9933.087134] pending: ptlrpc_pinger_main [ptlrpc] [ 9933.102074] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9923s workers=2 idle: 652 [ 9948.057105] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9948.060188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9948.113023] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9948.119667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9948.126753] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9948.132179] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9948.146309] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9948.161737] RAX: 0000000000006968 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9948.173241] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9948.179044] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9948.183118] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9948.186413] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9948.190823] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9948.195548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9948.198952] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9948.202336] Call Trace: [ 9948.203964] [ 9948.205411] ? show_regs.cold.9+0x22/0x2f [ 9948.209185] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9948.212181] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9948.215207] ? watchdog+0x40/0x40 [ 9948.218608] ? hrtimer_interrupt+0x128/0x2c0 [ 9948.224107] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9948.230534] ? apic_timer_interrupt+0xf/0x20 [ 9948.234014] [ 9948.235524] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9948.242554] do_raw_spin_lock+0x177/0x190 [ 9948.247864] _raw_spin_lock+0x21/0x30 [ 9948.250574] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9948.262621] ? do_raw_spin_unlock+0x75/0x190 [ 9948.269413] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9948.278104] ? do_wait_intr+0xf0/0xf0 [ 9948.279341] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9948.286732] mdc_close+0x53b/0xd70 [mdc] [ 9948.290137] lmv_close+0x18d/0x490 [lmv] [ 9948.295138] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9948.300105] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9948.305545] ll_md_close+0x274/0x800 [lustre] [ 9948.310129] ll_file_release+0x128/0xc50 [lustre] [ 9948.320110] __fput+0x10d/0x440 [ 9948.322499] ____fput+0x12/0x20 [ 9948.324725] task_work_run+0xc6/0x110 [ 9948.328271] exit_to_usermode_loop+0x1dd/0x1f0 [ 9948.330993] do_syscall_64+0x3d6/0x440 [ 9948.332253] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9948.339995] RIP: 0033:0x7f1a63983cd8 [ 9948.342167] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9948.352386] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9948.356185] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9948.360596] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9948.369997] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9948.375963] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9948.380605] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9960.056018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 9960.058028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 9960.059027] Modules linked in: [ 9960.060977] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9960.061074] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9960.061078] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9960.061081] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9960.061096] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9960.061099] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9960.061101] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9960.061103] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9960.061105] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9960.061106] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9960.061108] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9960.061111] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9960.061113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9960.061114] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9960.061121] Call Trace: [ 9960.061136] [ 9960.061141] ? show_regs.cold.9+0x22/0x2f [ 9960.061151] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9960.061157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9960.061161] ? watchdog+0x40/0x40 [ 9960.061166] ? hrtimer_interrupt+0x128/0x2c0 [ 9960.061170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9960.061177] ? apic_timer_interrupt+0xf/0x20 [ 9960.061181] [ 9960.061184] ? native_safe_halt+0x2e/0x50 [ 9960.061188] kvm_wait+0xcd/0xe0 [ 9960.061196] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9960.061204] do_raw_spin_lock+0x177/0x190 [ 9960.061209] _raw_spin_lock+0x21/0x30 [ 9960.061214] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9960.061475] obd_mod_free+0x46/0x140 [ptlrpc] [ 9960.061638] mdc_commit_open+0x91/0xb0 [mdc] [ 9960.061665] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9960.061788] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9960.061913] after_reply+0xe2c/0x1780 [ptlrpc] [ 9960.062038] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9960.062191] ? __next_timer_interrupt+0x160/0x160 [ 9960.062199] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9960.062389] ? do_wait_intr+0xf0/0xf0 [ 9960.062396] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9960.062581] kthread+0x1d1/0x200 [ 9960.062585] ? set_kthread_struct+0x70/0x70 [ 9960.062588] ret_from_fork+0x1f/0x30 [ 9960.064669] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 9960.064679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9960.064741] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9960.064745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9960.064747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9960.064759] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9960.064762] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9960.064766] RAX: 000000000000713e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9960.064768] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9960.064770] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9960.064773] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9960.064774] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9960.064777] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9960.064780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9960.064782] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9960.064789] Call Trace: [ 9960.064793] [ 9960.064797] ? show_regs.cold.9+0x22/0x2f [ 9960.064805] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9960.064810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9960.064814] ? watchdog+0x40/0x40 [ 9960.064818] ? hrtimer_interrupt+0x128/0x2c0 [ 9960.064822] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9960.064830] ? apic_timer_interrupt+0xf/0x20 [ 9960.064835] [ 9960.064838] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9960.064845] do_raw_spin_lock+0x177/0x190 [ 9960.064850] _raw_spin_lock+0x21/0x30 [ 9960.064855] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9960.067158] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9960.068063] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9960.068070] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9960.068073] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9960.068085] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9960.068089] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9960.068094] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9960.068096] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9960.068098] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9960.068100] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9960.068103] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9960.068106] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9960.068109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9960.068111] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9960.068119] Call Trace: [ 9960.068124] [ 9960.068129] ? show_regs.cold.9+0x22/0x2f [ 9960.068149] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9960.068155] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9960.068159] ? watchdog+0x40/0x40 [ 9960.068164] ? hrtimer_interrupt+0x128/0x2c0 [ 9960.068168] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9960.068175] ? apic_timer_interrupt+0xf/0x20 [ 9960.068179] [ 9960.068183] ? native_safe_halt+0x2e/0x50 [ 9960.068187] kvm_wait+0xcd/0xe0 [ 9960.068194] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9960.068203] do_raw_spin_lock+0x177/0x190 [ 9960.068207] _raw_spin_lock+0x21/0x30 [ 9960.068211] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9960.165741] ? do_raw_spin_unlock+0x75/0x190 [ 9960.182681] ? do_raw_spin_unlock+0x75/0x190 [ 9960.187237] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9960.187655] ? do_wait_intr+0xf0/0xf0 [ 9960.187666] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9960.188147] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9960.188683] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9960.188899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9960.189011] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9960.189048] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9960.189077] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9960.189270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9960.189365] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9960.189701] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9960.189728] lmv_intent_open+0x338/0x1150 [lmv] [ 9960.189751] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9960.189832] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9960.189917] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9960.189941] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9960.190057] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9960.190134] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9960.190274] ? do_raw_spin_unlock+0x75/0x190 [ 9960.190281] ? _raw_spin_unlock+0x12/0x30 [ 9960.190287] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9960.190308] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9960.190340] ll_atomic_open+0x36a/0xd70 [lustre] [ 9960.190423] lookup_open+0x356/0xb40 [ 9960.190431] do_last+0x313/0xfc0 [ 9960.190435] ? nd_jump_root+0xe5/0x160 [ 9960.190440] ? path_init+0x437/0x520 [ 9960.190444] path_openat+0xf7/0x500 [ 9960.190448] ? do_raw_spin_unlock+0x75/0x190 [ 9960.190452] do_filp_open+0x99/0x140 [ 9960.190457] ? getname_flags+0x6e/0x330 [ 9960.190460] ? __check_object_size+0xff/0x256 [ 9960.190465] ? do_raw_spin_unlock+0x75/0x190 [ 9960.190468] ? _raw_spin_unlock+0x12/0x30 [ 9960.190573] do_sys_openat2+0x2b4/0x410 [ 9960.190577] do_sys_open+0x73/0xa0 [ 9960.190581] __x64_sys_openat+0x24/0x30 [ 9960.190584] do_syscall_64+0xc1/0x440 [ 9960.190589] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9960.190595] RIP: 0033:0x7f1a63983332 [ 9960.190615] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9960.190619] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9960.190625] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9960.190627] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9960.190629] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9960.190631] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9960.190633] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9961.302103] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9961.316238] ? do_wait_intr+0xf0/0xf0 [ 9961.321139] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9961.334977] mdc_close+0x53b/0xd70 [mdc] [ 9961.345169] lmv_close+0x18d/0x490 [lmv] [ 9961.350749] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9961.359877] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9961.366286] ll_md_close+0x274/0x800 [lustre] [ 9961.368005] ll_file_release+0x128/0xc50 [lustre] [ 9961.382158] __fput+0x10d/0x440 [ 9961.390928] ____fput+0x12/0x20 [ 9961.398776] task_work_run+0xc6/0x110 [ 9961.406689] exit_to_usermode_loop+0x1dd/0x1f0 [ 9961.409939] do_syscall_64+0x3d6/0x440 [ 9961.416040] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9961.422965] RIP: 0033:0x7f1a63983cd8 [ 9961.427000] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9961.451621] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9961.460774] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9961.470935] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9961.478224] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9961.486454] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9961.494227] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9963.489049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7940s! [ 9963.494077] Showing busy workqueues and worker pools: [ 9963.501678] workqueue events: flags=0x0 [ 9963.503774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9963.516059] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9963.521222] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9963.527788] pending: sec_gc_main [ptlrpc] [ 9963.531022] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9963.539712] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9963.550211] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9963.556383] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9963.563903] workqueue events_long: flags=0x0 [ 9963.566708] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9963.582002] pending: lu_objects_flush [obdclass] [ 9963.590209] workqueue events_unbound: flags=0x2 [ 9963.596115] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9963.599943] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9963.607062] workqueue events_power_efficient: flags=0x80 [ 9963.612916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9963.616192] pending: neigh_periodic_work, do_cache_clean [ 9963.619733] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9963.624432] pending: neigh_periodic_work, check_lifetime [ 9963.629276] workqueue mm_percpu_wq: flags=0x8 [ 9963.639815] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9963.652066] pending: vmstat_update [ 9963.656125] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9963.668713] pending: vmstat_update [ 9963.673912] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9963.682834] pending: vmstat_update [ 9963.686438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9963.694681] pending: vmstat_update [ 9963.697524] workqueue writeback: flags=0x4a [ 9963.701105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9963.708925] pending: wb_workfn, wb_workfn [ 9963.713175] workqueue xprtiod: flags=0xa [ 9963.716787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9963.723056] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9963.738884] workqueue ipv6_addrconf: flags=0x40008 [ 9963.742160] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9963.755849] pending: addrconf_verify_work [ 9963.766169] workqueue knbd0-recv: flags=0x1a [ 9963.773926] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9963.783899] in-flight: 647:recv_work [ 9963.789057] workqueue ptlrpc_pinger: flags=0x2 [ 9963.791928] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9963.802453] pending: ptlrpc_pinger_main [ptlrpc] [ 9963.806374] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9953s workers=2 idle: 652 [ 9976.060242] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [ 9976.064899] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9976.107252] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9976.117262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9976.125238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9976.128992] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9976.141281] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9976.145702] RAX: 0000000000007492 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9976.152672] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [ 9976.156051] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [ 9976.160790] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [ 9976.170023] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [ 9976.174230] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [ 9976.181986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9976.184373] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [ 9976.187716] Call Trace: [ 9976.188805] [ 9976.189563] ? show_regs.cold.9+0x22/0x2f [ 9976.193881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9976.196938] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9976.198813] ? watchdog+0x40/0x40 [ 9976.200123] ? hrtimer_interrupt+0x128/0x2c0 [ 9976.205253] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9976.207752] ? apic_timer_interrupt+0xf/0x20 [ 9976.210176] [ 9976.211103] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9976.213391] do_raw_spin_lock+0x177/0x190 [ 9976.215272] _raw_spin_lock+0x21/0x30 [ 9976.216745] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9976.223583] ? do_raw_spin_unlock+0x75/0x190 [ 9976.228804] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9976.230421] ? do_wait_intr+0xf0/0xf0 [ 9976.233012] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9976.238785] mdc_close+0x53b/0xd70 [mdc] [ 9976.242311] lmv_close+0x18d/0x490 [lmv] [ 9976.247279] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9976.250106] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9976.253012] ll_md_close+0x274/0x800 [lustre] [ 9976.254853] ll_file_release+0x128/0xc50 [lustre] [ 9976.258109] __fput+0x10d/0x440 [ 9976.263526] ____fput+0x12/0x20 [ 9976.265245] task_work_run+0xc6/0x110 [ 9976.267076] exit_to_usermode_loop+0x1dd/0x1f0 [ 9976.269569] do_syscall_64+0x3d6/0x440 [ 9976.271287] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9976.273668] RIP: 0033:0x7f1a63983cd8 [ 9976.275577] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9976.283778] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9976.289881] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [ 9976.294851] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9976.303474] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [ 9976.305769] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [ 9976.311129] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9988.054079] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [ 9988.057887] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [ 9988.061129] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [ 9988.061142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9988.061219] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9988.061225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9988.061228] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9988.061243] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9988.061248] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9988.061401] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [ 9988.061403] RDX: 0000000000007818 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [ 9988.061406] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [ 9988.061408] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [ 9988.061411] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9988.061414] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [ 9988.061417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9988.061419] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [ 9988.061428] Call Trace: [ 9988.061433] [ 9988.061438] ? show_regs.cold.9+0x22/0x2f [ 9988.061447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9988.061453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9988.061457] ? watchdog+0x40/0x40 [ 9988.061461] ? hrtimer_interrupt+0x128/0x2c0 [ 9988.061465] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9988.061474] ? apic_timer_interrupt+0xf/0x20 [ 9988.061479] [ 9988.061482] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9988.061487] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9988.061492] do_raw_spin_lock+0x177/0x190 [ 9988.061497] _raw_spin_lock+0x21/0x30 [ 9988.061503] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9988.062226] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [ 9988.062239] obd_mod_free+0x46/0x140 [ptlrpc] [ 9988.062287] sha512_ssse3 sha512_generic [ 9988.062739] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9988.062745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9988.062747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9988.062756] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9988.062760] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9988.062765] RAX: 000000000000526c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9988.062767] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [ 9988.062770] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [ 9988.062772] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [ 9988.062774] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [ 9988.062777] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [ 9988.062780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9988.062783] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [ 9988.062789] Call Trace: [ 9988.062792] [ 9988.062796] ? show_regs.cold.9+0x22/0x2f [ 9988.062804] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9988.062810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9988.062814] ? watchdog+0x40/0x40 [ 9988.062818] ? hrtimer_interrupt+0x128/0x2c0 [ 9988.062822] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9988.062828] ? apic_timer_interrupt+0xf/0x20 [ 9988.062833] [ 9988.062836] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9988.062842] do_raw_spin_lock+0x177/0x190 [ 9988.062846] _raw_spin_lock+0x21/0x30 [ 9988.063192] mdc_commit_open+0x91/0xb0 [mdc] [ 9988.062851] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9988.063493] ? do_raw_spin_unlock+0x75/0x190 [ 9988.063500] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9988.063797] ? do_wait_intr+0xf0/0xf0 [ 9988.063806] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9988.065827] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9988.066305] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9988.066617] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9988.067463] after_reply+0xe2c/0x1780 [ptlrpc] [ 9988.067667] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9988.068058] ? __next_timer_interrupt+0x160/0x160 [ 9988.068068] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9988.068668] ? do_wait_intr+0xf0/0xf0 [ 9988.068678] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9988.070465] kthread+0x1d1/0x200 [ 9988.070474] ? set_kthread_struct+0x70/0x70 [ 9988.070480] ret_from_fork+0x1f/0x30 [ 9988.074451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9988.074536] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9988.074545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9988.074548] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9988.074562] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9988.074566] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9988.074571] RAX: 0000000000000d9c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [ 9988.074574] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [ 9988.074576] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9988.074578] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [ 9988.074581] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [ 9988.074585] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [ 9988.074588] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9988.074591] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [ 9988.074600] Call Trace: [ 9988.074605] [ 9988.074610] ? show_regs.cold.9+0x22/0x2f [ 9988.074620] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9988.074626] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9988.074630] ? watchdog+0x40/0x40 [ 9988.074635] ? hrtimer_interrupt+0x128/0x2c0 [ 9988.074638] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9988.074647] ? apic_timer_interrupt+0xf/0x20 [ 9988.074653] [ 9988.074655] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9988.074661] do_raw_spin_lock+0x177/0x190 [ 9988.074666] _raw_spin_lock+0x21/0x30 [ 9988.074671] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9988.079496] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9988.080431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9988.080696] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9988.080745] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9988.080789] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9988.080828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9988.081456] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9988.126492] ? do_raw_spin_unlock+0x75/0x190 [ 9988.126522] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9988.126875] ? do_wait_intr+0xf0/0xf0 [ 9988.126883] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9988.127142] mdc_close+0x53b/0xd70 [mdc] [ 9988.127193] lmv_close+0x18d/0x490 [lmv] [ 9988.127228] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9988.127614] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9988.127704] ll_md_close+0x274/0x800 [lustre] [ 9988.127787] ll_file_release+0x128/0xc50 [lustre] [ 9988.127871] __fput+0x10d/0x440 [ 9988.127881] ____fput+0x12/0x20 [ 9988.127886] task_work_run+0xc6/0x110 [ 9988.127892] exit_to_usermode_loop+0x1dd/0x1f0 [ 9988.127898] do_syscall_64+0x3d6/0x440 [ 9988.127904] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9988.127913] RIP: 0033:0x7f1a63983cd8 [ 9988.127920] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9988.127924] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9988.127929] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [ 9988.127932] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [ 9988.127934] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [ 9988.127936] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [ 9988.127938] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9988.648801] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9988.651494] lmv_intent_open+0x338/0x1150 [lmv] [ 9988.653769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9988.656497] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9988.659531] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9988.661777] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9988.664490] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9988.667350] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9988.669527] ? do_raw_spin_unlock+0x75/0x190 [ 9988.671553] ? _raw_spin_unlock+0x12/0x30 [ 9988.673459] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9988.676648] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9988.679259] ll_atomic_open+0x36a/0xd70 [lustre] [ 9988.681582] lookup_open+0x356/0xb40 [ 9988.683556] do_last+0x313/0xfc0 [ 9988.684818] ? nd_jump_root+0xe5/0x160 [ 9988.686459] ? path_init+0x437/0x520 [ 9988.688032] path_openat+0xf7/0x500 [ 9988.689875] ? do_raw_spin_unlock+0x75/0x190 [ 9988.692788] do_filp_open+0x99/0x140 [ 9988.694378] ? getname_flags+0x6e/0x330 [ 9988.696569] ? __check_object_size+0xff/0x256 [ 9988.698510] ? do_raw_spin_unlock+0x75/0x190 [ 9988.700853] ? _raw_spin_unlock+0x12/0x30 [ 9988.702637] do_sys_openat2+0x2b4/0x410 [ 9988.704496] do_sys_open+0x73/0xa0 [ 9988.705909] __x64_sys_openat+0x24/0x30 [ 9988.707579] do_syscall_64+0xc1/0x440 [ 9988.709661] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9988.712687] RIP: 0033:0x7f1a63983332 [ 9988.714834] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9988.723944] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9988.728320] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [ 9988.732193] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [ 9988.736554] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [ 9988.740831] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [ 9988.744691] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 9994.210331] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7971s! [ 9994.226166] Showing busy workqueues and worker pools: [ 9994.236322] workqueue events: flags=0x0 [ 9994.242210] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9994.247801] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [ 9994.254240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9994.266822] pending: sec_gc_main [ptlrpc] [ 9994.275936] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9994.287465] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [ 9994.306293] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9994.317873] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [ 9994.335064] workqueue events_long: flags=0x0 [ 9994.341935] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9994.361639] pending: lu_objects_flush [obdclass] [ 9994.381360] workqueue events_unbound: flags=0x2 [ 9994.388051] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9994.395706] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9994.412551] workqueue events_power_efficient: flags=0x80 [ 9994.416806] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9994.420749] pending: neigh_periodic_work, do_cache_clean [ 9994.426839] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9994.435770] pending: neigh_periodic_work, check_lifetime [ 9994.455348] workqueue mm_percpu_wq: flags=0x8 [ 9994.464977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9994.475688] pending: vmstat_update [ 9994.482697] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9994.490219] pending: vmstat_update [ 9994.501817] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9994.512274] pending: vmstat_update [ 9994.513475] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9994.521958] pending: vmstat_update [ 9994.524262] workqueue writeback: flags=0x4a [ 9994.529884] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9994.531705] pending: wb_workfn, wb_workfn [ 9994.537737] workqueue xprtiod: flags=0xa [ 9994.540197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9994.547205] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [ 9994.554964] workqueue ipv6_addrconf: flags=0x40008 [ 9994.560818] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9994.566308] pending: addrconf_verify_work [ 9994.569657] workqueue knbd0-recv: flags=0x1a [ 9994.573386] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9994.576264] in-flight: 647:recv_work [ 9994.580101] workqueue ptlrpc_pinger: flags=0x2 [ 9994.584742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9994.588591] pending: ptlrpc_pinger_main [ptlrpc] [ 9994.591619] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9984s workers=2 idle: 652 [10002.625687] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [10002.638008] rcu: 3-...!: (5039680 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [10002.645517] (detected by 1, t=7980219 jiffies, g=269633, q=2865) [10002.653749] Sending NMI from CPU 1 to CPUs 3: [10002.661737] NMI backtrace for cpu 3 [10002.661745] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10002.661747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10002.661748] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10002.661750] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10002.661752] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [10002.661755] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10002.661756] RDX: 000000000000125a RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10002.661757] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10002.661759] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10002.661761] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10002.661762] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10002.661763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10002.661765] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10002.661766] Call Trace: [10002.661767] [10002.661768] ? show_regs.cold.9+0x22/0x2f [10002.661769] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10002.661770] ? nmi_cpu_backtrace_handler+0x15/0x40 [10002.661772] ? nmi_handle+0xbd/0x210 [10002.661773] ? default_do_nmi+0x4b/0x1b0 [10002.661774] ? do_nmi+0x280/0x380 [10002.661775] ? end_repeat_nmi+0x16/0x1f [10002.661776] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10002.661777] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10002.661779] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10002.661780] [10002.661781] do_raw_spin_lock+0x177/0x190 [10002.661782] _raw_spin_lock+0x21/0x30 [10002.661783] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10002.661784] obd_mod_free+0x46/0x140 [ptlrpc] [10002.661785] mdc_commit_open+0x91/0xb0 [mdc] [10002.661786] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10002.661787] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10002.661789] after_reply+0xe2c/0x1780 [ptlrpc] [10002.661790] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10002.661792] ? __next_timer_interrupt+0x160/0x160 [10002.661793] ptlrpcd+0xa82/0xe40 [ptlrpc] [10002.661794] ? do_wait_intr+0xf0/0xf0 [10002.661795] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10002.661797] kthread+0x1d1/0x200 [10002.661798] ? set_kthread_struct+0x70/0x70 [10002.661799] ret_from_fork+0x1f/0x30 [10002.663659] rcu: rcu_sched kthread starved for 7979865 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [10003.106988] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10003.126769] rcu: RCU grace-period kthread stack dump: [10003.139880] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10003.159888] Call Trace: [10003.169726] __schedule+0x351/0xcb0 [10003.173358] schedule+0xc0/0x180 [10003.182558] schedule_timeout+0xb4/0x190 [10003.190878] ? do_raw_spin_unlock+0x75/0x190 [10003.206288] ? __next_timer_interrupt+0x160/0x160 [10003.227014] ? prepare_to_swait_event+0xf3/0x140 [10003.231974] rcu_gp_kthread+0x8dd/0x15b0 [10003.244423] ? force_qs_rnp+0x2c0/0x2c0 [10003.249541] kthread+0x1d1/0x200 [10003.258394] ? set_kthread_struct+0x70/0x70 [10003.268438] ret_from_fork+0x1f/0x30 [10003.274243] rcu: Stack dump where RCU GP kthread last ran: [10003.285019] Sending NMI from CPU 1 to CPUs 0: [10003.299885] NMI backtrace for cpu 0 [10003.299891] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10003.299893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10003.299895] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10003.299896] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10003.299898] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 [10003.299910] RAX: 0000000000007208 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10003.299912] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10003.299916] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10003.299918] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10003.299919] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10003.299921] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10003.299923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10003.299924] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10003.299925] Call Trace: [10003.299926] [10003.299927] ? show_regs.cold.9+0x22/0x2f [10003.299928] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10003.299929] ? nmi_cpu_backtrace_handler+0x15/0x40 [10003.299930] ? nmi_handle+0xbd/0x210 [10003.299931] ? default_do_nmi+0x4b/0x1b0 [10003.299931] ? do_nmi+0x280/0x380 [10003.299933] ? end_repeat_nmi+0x16/0x1f [10003.299934] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10003.299938] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10003.299942] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10003.299944] [10003.299945] do_raw_spin_lock+0x177/0x190 [10003.299945] _raw_spin_lock+0x21/0x30 [10003.299946] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10003.299948] ? do_raw_spin_unlock+0x75/0x190 [10003.299949] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10003.299950] ? do_wait_intr+0xf0/0xf0 [10003.299951] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10003.299952] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10003.299953] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10003.299955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10003.299956] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10003.299957] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10003.299958] mdc_intent_lock+0x353/0x7e0 [mdc] [10003.299959] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10003.299960] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10003.299961] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10003.299963] lmv_intent_open+0x338/0x1150 [lmv] [10003.299964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10003.299965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10003.299966] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10003.299967] ll_intent_lock+0xc5/0xaa0 [lustre] [10003.299968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10003.299969] ll_lookup_it+0xe0e/0x2270 [lustre] [10003.299970] ? do_raw_spin_unlock+0x75/0x190 [10003.299971] ? _raw_spin_unlock+0x12/0x30 [10003.299972] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10003.299973] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10003.299974] ll_atomic_open+0x36a/0xd70 [lustre] [10003.299975] lookup_open+0x356/0xb40 [10003.299976] do_last+0x313/0xfc0 [10003.299977] ? nd_jump_root+0xe5/0x160 [10003.299978] ? path_init+0x437/0x520 [10003.299979] path_openat+0xf7/0x500 [10003.299979] ? do_raw_spin_unlock+0x75/0x190 [10003.299980] do_filp_open+0x99/0x140 [10003.299982] ? getname_flags+0x6e/0x330 [10003.299982] ? __check_object_size+0xff/0x256 [10003.299983] ? do_raw_spin_unlock+0x75/0x190 [10003.299984] ? _raw_spin_unlock+0x12/0x30 [10003.299985] do_sys_openat2+0x2b4/0x410 [10003.299986] do_sys_open+0x73/0xa0 [10003.299987] __x64_sys_openat+0x24/0x30 [10003.299988] do_syscall_64+0xc1/0x440 [10003.299989] entry_SYSCALL_64_after_hwframe+0x49/0xae [10003.299990] RIP: 0033:0x7f1a63983332 [10003.299992] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10003.299994] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10003.299996] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10003.299997] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10003.299999] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10003.300000] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10003.300001] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10004.058552] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10004.068160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10004.097296] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10004.110029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10004.115905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10004.119877] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10004.133664] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10004.138632] RAX: 0000000000005208 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10004.145441] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10004.150358] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10004.155647] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10004.159850] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10004.165537] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10004.171201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10004.174658] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10004.179317] Call Trace: [10004.181361] [10004.182452] ? show_regs.cold.9+0x22/0x2f [10004.185113] ? watchdog_timer_fn.cold.6+0x62/0x10a [10004.188600] ? __hrtimer_run_queues+0x1c2/0x4d0 [10004.191854] ? watchdog+0x40/0x40 [10004.193706] ? hrtimer_interrupt+0x128/0x2c0 [10004.196239] ? smp_apic_timer_interrupt+0xa6/0x270 [10004.199051] ? apic_timer_interrupt+0xf/0x20 [10004.202285] [10004.203527] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10004.206826] do_raw_spin_lock+0x177/0x190 [10004.209554] _raw_spin_lock+0x21/0x30 [10004.212051] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10004.222843] ? do_raw_spin_unlock+0x75/0x190 [10004.225671] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10004.228886] ? do_wait_intr+0xf0/0xf0 [10004.232375] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10004.235644] mdc_close+0x53b/0xd70 [mdc] [10004.240194] lmv_close+0x18d/0x490 [lmv] [10004.244567] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10004.250445] ll_md_real_close+0xa5/0x2a0 [lustre] [10004.257788] ll_md_close+0x274/0x800 [lustre] [10004.263557] ll_file_release+0x128/0xc50 [lustre] [10004.270769] __fput+0x10d/0x440 [10004.272773] ____fput+0x12/0x20 [10004.274648] task_work_run+0xc6/0x110 [10004.278158] exit_to_usermode_loop+0x1dd/0x1f0 [10004.281425] do_syscall_64+0x3d6/0x440 [10004.285425] entry_SYSCALL_64_after_hwframe+0x49/0xae [10004.296445] RIP: 0033:0x7f1a63983cd8 [10004.300062] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10004.318563] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10004.325106] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10004.329776] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10004.335253] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10004.344888] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10004.350546] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10024.933741] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8001s! [10024.936562] Showing busy workqueues and worker pools: [10024.940023] workqueue events: flags=0x0 [10024.942886] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10024.946164] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10024.949212] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10024.953091] pending: sec_gc_main [ptlrpc] [10024.955579] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10024.958630] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10024.963836] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10024.967654] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10024.972601] workqueue events_long: flags=0x0 [10024.974794] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10024.979084] pending: lu_objects_flush [obdclass] [10024.982830] workqueue events_unbound: flags=0x2 [10024.986464] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10024.992842] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10024.998639] workqueue events_power_efficient: flags=0x80 [10025.001749] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10025.006673] pending: neigh_periodic_work, do_cache_clean [10025.009877] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10025.013836] pending: neigh_periodic_work, check_lifetime [10025.017266] workqueue mm_percpu_wq: flags=0x8 [10025.020035] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10025.024814] pending: vmstat_update [10025.027764] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10025.033329] pending: vmstat_update [10025.036119] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10025.040444] pending: vmstat_update [10025.042655] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10025.046782] pending: vmstat_update [10025.048918] workqueue writeback: flags=0x4a [10025.050266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10025.052541] pending: wb_workfn, wb_workfn [10025.055141] workqueue xprtiod: flags=0xa [10025.058322] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10025.061695] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10025.068156] workqueue ipv6_addrconf: flags=0x40008 [10025.073455] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10025.076617] pending: addrconf_verify_work [10025.079019] workqueue knbd0-recv: flags=0x1a [10025.081339] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10025.087844] in-flight: 647:recv_work [10025.089565] workqueue ptlrpc_pinger: flags=0x2 [10025.094528] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10025.103133] pending: ptlrpc_pinger_main [ptlrpc] [10025.113328] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10015s workers=2 idle: 652 [10028.056033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10028.058889] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10028.064712] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10028.064790] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10028.064797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10028.064800] RIP: 0010:native_safe_halt+0x2e/0x50 [10028.064816] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10028.064820] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10028.064825] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10028.064827] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10028.064829] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10028.064831] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10028.064833] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10028.064837] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10028.064840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10028.064842] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10028.064862] Call Trace: [10028.064868] [10028.064874] ? show_regs.cold.9+0x22/0x2f [10028.064892] ? watchdog_timer_fn.cold.6+0x62/0x10a [10028.064898] ? __hrtimer_run_queues+0x1c2/0x4d0 [10028.064902] ? watchdog+0x40/0x40 [10028.064912] ? hrtimer_interrupt+0x128/0x2c0 [10028.064920] ? smp_apic_timer_interrupt+0xa6/0x270 [10028.064927] ? apic_timer_interrupt+0xf/0x20 [10028.064932] [10028.064935] ? native_safe_halt+0x2e/0x50 [10028.064939] kvm_wait+0xcd/0xe0 [10028.064946] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10028.064954] do_raw_spin_lock+0x177/0x190 [10028.064959] _raw_spin_lock+0x21/0x30 [10028.064963] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10028.074710] Modules linked in: [10028.130138] ? do_raw_spin_unlock+0x75/0x190 [10028.130171] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10028.130464] ? do_wait_intr+0xf0/0xf0 [10028.130472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10028.130833] mdc_close+0x53b/0xd70 [mdc] [10028.130885] lmv_close+0x18d/0x490 [lmv] [10028.130916] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10028.131025] ll_md_real_close+0xa5/0x2a0 [lustre] [10028.131105] ll_md_close+0x274/0x800 [lustre] [10028.131204] ll_file_release+0x128/0xc50 [lustre] [10028.131569] __fput+0x10d/0x440 [10028.131578] ____fput+0x12/0x20 [10028.131583] task_work_run+0xc6/0x110 [10028.131590] exit_to_usermode_loop+0x1dd/0x1f0 [10028.131597] do_syscall_64+0x3d6/0x440 [10028.131602] entry_SYSCALL_64_after_hwframe+0x49/0xae [10028.131611] RIP: 0033:0x7f1a63983cd8 [10028.131619] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10028.131623] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10028.131628] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10028.131631] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10028.131633] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10028.131635] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10028.131637] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10028.381861] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10028.413526] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10028.422016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10028.433213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10028.439855] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10028.451064] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10028.458519] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10028.463905] RDX: 00000000000039c5 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10028.466405] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10028.469638] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10028.473240] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10028.480519] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10028.489200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10028.493217] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10028.502364] Call Trace: [10028.504073] [10028.505827] ? show_regs.cold.9+0x22/0x2f [10028.508353] ? watchdog_timer_fn.cold.6+0x62/0x10a [10028.516541] ? __hrtimer_run_queues+0x1c2/0x4d0 [10028.524169] ? watchdog+0x40/0x40 [10028.527110] ? hrtimer_interrupt+0x128/0x2c0 [10028.533104] ? smp_apic_timer_interrupt+0xa6/0x270 [10028.537201] ? apic_timer_interrupt+0xf/0x20 [10028.539301] [10028.542694] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10028.547732] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10028.552716] do_raw_spin_lock+0x177/0x190 [10028.560130] _raw_spin_lock+0x21/0x30 [10028.564203] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10028.569780] obd_mod_free+0x46/0x140 [ptlrpc] [10028.571150] mdc_commit_open+0x91/0xb0 [mdc] [10028.573323] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10028.576631] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10028.580348] after_reply+0xe2c/0x1780 [ptlrpc] [10028.585198] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10028.587980] ? __next_timer_interrupt+0x160/0x160 [10028.592484] ptlrpcd+0xa82/0xe40 [ptlrpc] [10028.597214] ? do_wait_intr+0xf0/0xf0 [10028.599101] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10028.607087] kthread+0x1d1/0x200 [10028.612372] ? set_kthread_struct+0x70/0x70 [10028.616833] ret_from_fork+0x1f/0x30 [10032.057316] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10032.063186] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10032.110678] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10032.127512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10032.139679] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10032.143162] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10032.161347] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10032.168139] RAX: 0000000000007d15 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10032.170190] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10032.178991] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10032.182534] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10032.188571] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10032.192582] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10032.195837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10032.198147] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10032.201446] Call Trace: [10032.204018] [10032.206328] ? show_regs.cold.9+0x22/0x2f [10032.208782] ? watchdog_timer_fn.cold.6+0x62/0x10a [10032.212305] ? __hrtimer_run_queues+0x1c2/0x4d0 [10032.214286] ? watchdog+0x40/0x40 [10032.216418] ? hrtimer_interrupt+0x128/0x2c0 [10032.218881] ? smp_apic_timer_interrupt+0xa6/0x270 [10032.222104] ? apic_timer_interrupt+0xf/0x20 [10032.224385] [10032.225500] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10032.228535] do_raw_spin_lock+0x177/0x190 [10032.230331] _raw_spin_lock+0x21/0x30 [10032.232328] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10032.234841] ? do_raw_spin_unlock+0x75/0x190 [10032.236763] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10032.239567] ? do_wait_intr+0xf0/0xf0 [10032.241375] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10032.243829] mdc_close+0x53b/0xd70 [mdc] [10032.246009] lmv_close+0x18d/0x490 [lmv] [10032.248031] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10032.250842] ll_md_real_close+0xa5/0x2a0 [lustre] [10032.260252] ll_md_close+0x274/0x800 [lustre] [10032.263049] ll_file_release+0x128/0xc50 [lustre] [10032.266005] __fput+0x10d/0x440 [10032.276565] ____fput+0x12/0x20 [10032.279528] task_work_run+0xc6/0x110 [10032.282446] exit_to_usermode_loop+0x1dd/0x1f0 [10032.284637] do_syscall_64+0x3d6/0x440 [10032.287206] entry_SYSCALL_64_after_hwframe+0x49/0xae [10032.290021] RIP: 0033:0x7f1a63983cd8 [10032.292079] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10032.308515] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10032.315538] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10032.319835] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10032.322613] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10032.325451] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10032.328390] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10052.053029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10052.055561] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10052.078160] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10052.081998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10052.085219] RIP: 0010:native_safe_halt+0x2e/0x50 [10052.087538] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10052.097179] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10052.101134] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10052.104961] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [10052.108116] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10052.111214] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10052.115232] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10052.118621] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10052.122912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10052.125874] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10052.129774] Call Trace: [10052.131516] [10052.132743] ? show_regs.cold.9+0x22/0x2f [10052.134197] ? watchdog_timer_fn.cold.6+0x62/0x10a [10052.138127] ? __hrtimer_run_queues+0x1c2/0x4d0 [10052.141165] ? watchdog+0x40/0x40 [10052.143240] ? hrtimer_interrupt+0x128/0x2c0 [10052.146590] ? smp_apic_timer_interrupt+0xa6/0x270 [10052.149281] ? apic_timer_interrupt+0xf/0x20 [10052.152122] [10052.153318] ? native_safe_halt+0x2e/0x50 [10052.156084] kvm_wait+0xcd/0xe0 [10052.158084] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10052.161638] do_raw_spin_lock+0x177/0x190 [10052.164183] _raw_spin_lock+0x21/0x30 [10052.166620] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10052.170224] ? do_raw_spin_unlock+0x75/0x190 [10052.173045] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10052.176954] ? do_wait_intr+0xf0/0xf0 [10052.179240] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10052.181957] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10052.184320] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10052.187102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10052.189408] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10052.191884] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10052.193757] mdc_intent_lock+0x353/0x7e0 [mdc] [10052.195961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10052.197957] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10052.201914] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10052.205240] lmv_intent_open+0x338/0x1150 [lmv] [10052.207946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10052.211402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10052.214845] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10052.217179] ll_intent_lock+0xc5/0xaa0 [lustre] [10052.219913] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10052.223610] ll_lookup_it+0xe0e/0x2270 [lustre] [10052.226060] ? do_raw_spin_unlock+0x75/0x190 [10052.228277] ? _raw_spin_unlock+0x12/0x30 [10052.230566] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10052.233310] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10052.237171] ll_atomic_open+0x36a/0xd70 [lustre] [10052.242870] lookup_open+0x356/0xb40 [10052.245503] do_last+0x313/0xfc0 [10052.247004] ? nd_jump_root+0xe5/0x160 [10052.249661] ? path_init+0x437/0x520 [10052.251485] path_openat+0xf7/0x500 [10052.253164] ? do_raw_spin_unlock+0x75/0x190 [10052.255479] do_filp_open+0x99/0x140 [10052.257515] ? getname_flags+0x6e/0x330 [10052.259203] ? __check_object_size+0xff/0x256 [10052.262196] ? do_raw_spin_unlock+0x75/0x190 [10052.264723] ? _raw_spin_unlock+0x12/0x30 [10052.267160] do_sys_openat2+0x2b4/0x410 [10052.269390] do_sys_open+0x73/0xa0 [10052.271172] __x64_sys_openat+0x24/0x30 [10052.274368] do_syscall_64+0xc1/0x440 [10052.276626] entry_SYSCALL_64_after_hwframe+0x49/0xae [10052.281645] RIP: 0033:0x7f1a63983332 [10052.284047] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10052.294881] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10052.299336] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10052.303436] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10052.307861] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10052.312513] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10052.317348] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10055.136504] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8032s! [10055.140073] Showing busy workqueues and worker pools: [10055.147167] workqueue events: flags=0x0 [10055.150894] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10055.164897] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10055.179753] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10055.185269] pending: sec_gc_main [ptlrpc] [10055.188075] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10055.197855] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10055.207109] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10055.215226] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10055.227957] workqueue events_long: flags=0x0 [10055.232876] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10055.236846] pending: lu_objects_flush [obdclass] [10055.243856] workqueue events_unbound: flags=0x2 [10055.254208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10055.260285] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10055.269782] workqueue events_power_efficient: flags=0x80 [10055.272634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10055.275119] pending: neigh_periodic_work, do_cache_clean [10055.280605] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10055.283924] pending: neigh_periodic_work, check_lifetime [10055.286870] workqueue mm_percpu_wq: flags=0x8 [10055.288161] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10055.292579] pending: vmstat_update [10055.294266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10055.298735] pending: vmstat_update [10055.300310] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10055.304583] pending: vmstat_update [10055.309643] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10055.313265] pending: vmstat_update [10055.315224] workqueue writeback: flags=0x4a [10055.316324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10055.320620] pending: wb_workfn, wb_workfn [10055.322827] workqueue xprtiod: flags=0xa [10055.324971] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10055.329210] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10055.335616] workqueue ipv6_addrconf: flags=0x40008 [10055.338020] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10055.342717] pending: addrconf_verify_work [10055.345199] workqueue knbd0-recv: flags=0x1a [10055.347623] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10055.352169] in-flight: 647:recv_work [10055.355239] workqueue ptlrpc_pinger: flags=0x2 [10055.362278] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10055.366249] pending: ptlrpc_pinger_main [ptlrpc] [10055.368831] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10045s workers=2 idle: 652 [10056.056010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10056.058647] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10056.060218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10056.060295] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10056.060302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10056.060306] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [10056.060321] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [10056.060325] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10056.060330] RAX: 000000000000385d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10056.061028] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10056.061032] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10056.061035] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10056.061037] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10056.061040] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10056.061044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10056.061046] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10056.061054] Call Trace: [10056.061061] [10056.061066] ? show_regs.cold.9+0x22/0x2f [10056.061077] ? watchdog_timer_fn.cold.6+0x62/0x10a [10056.061083] ? __hrtimer_run_queues+0x1c2/0x4d0 [10056.061087] ? watchdog+0x40/0x40 [10056.061092] ? hrtimer_interrupt+0x128/0x2c0 [10056.061096] ? smp_apic_timer_interrupt+0xa6/0x270 [10056.061104] ? apic_timer_interrupt+0xf/0x20 [10056.061109] [10056.061112] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [10056.061118] do_raw_spin_lock+0x177/0x190 [10056.061123] _raw_spin_lock+0x21/0x30 [10056.061128] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10056.061432] ? do_raw_spin_unlock+0x75/0x190 [10056.061438] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10056.061637] ? do_wait_intr+0xf0/0xf0 [10056.061646] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10056.061839] mdc_close+0x53b/0xd70 [mdc] [10056.061877] lmv_close+0x18d/0x490 [lmv] [10056.061904] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10056.062015] ll_md_real_close+0xa5/0x2a0 [lustre] [10056.062093] ll_md_close+0x274/0x800 [lustre] [10056.062175] ll_file_release+0x128/0xc50 [lustre] [10056.062256] __fput+0x10d/0x440 [10056.062266] ____fput+0x12/0x20 [10056.062270] task_work_run+0xc6/0x110 [10056.062277] exit_to_usermode_loop+0x1dd/0x1f0 [10056.062285] do_syscall_64+0x3d6/0x440 [10056.062290] entry_SYSCALL_64_after_hwframe+0x49/0xae [10056.062299] RIP: 0033:0x7f1a63983cd8 [10056.062306] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10056.062310] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10056.062314] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10056.062316] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10056.062319] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10056.062320] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10056.062322] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10056.210174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10056.229199] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10056.234253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10056.236932] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10056.239510] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10056.246935] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10056.249733] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10056.252532] RDX: 0000000000003e04 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10056.255370] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10056.257706] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10056.260211] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10056.263167] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10056.265897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10056.267777] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10056.270141] Call Trace: [10056.271030] [10056.271825] ? show_regs.cold.9+0x22/0x2f [10056.273793] ? watchdog_timer_fn.cold.6+0x62/0x10a [10056.275637] ? __hrtimer_run_queues+0x1c2/0x4d0 [10056.277041] ? watchdog+0x40/0x40 [10056.278159] ? hrtimer_interrupt+0x128/0x2c0 [10056.279858] ? smp_apic_timer_interrupt+0xa6/0x270 [10056.284231] ? apic_timer_interrupt+0xf/0x20 [10056.285900] [10056.288040] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10056.289911] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10056.292668] do_raw_spin_lock+0x177/0x190 [10056.294026] _raw_spin_lock+0x21/0x30 [10056.295182] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10056.297685] obd_mod_free+0x46/0x140 [ptlrpc] [10056.300106] mdc_commit_open+0x91/0xb0 [mdc] [10056.301779] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10056.303720] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10056.307237] after_reply+0xe2c/0x1780 [ptlrpc] [10056.310677] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10056.314540] ? __next_timer_interrupt+0x160/0x160 [10056.317018] ptlrpcd+0xa82/0xe40 [ptlrpc] [10056.320107] ? do_wait_intr+0xf0/0xf0 [10056.323727] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10056.327450] kthread+0x1d1/0x200 [10056.330661] ? set_kthread_struct+0x70/0x70 [10056.332998] ret_from_fork+0x1f/0x30 [10060.057021] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10060.059969] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10060.080190] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10060.085231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10060.089499] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10060.092264] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10060.100744] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10060.104378] RAX: 000000000000340c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10060.107542] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10060.110469] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10060.113850] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10060.116786] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10060.118866] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10060.121139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10060.123104] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10060.125399] Call Trace: [10060.126259] [10060.127122] ? show_regs.cold.9+0x22/0x2f [10060.128834] ? watchdog_timer_fn.cold.6+0x62/0x10a [10060.130857] ? __hrtimer_run_queues+0x1c2/0x4d0 [10060.132881] ? watchdog+0x40/0x40 [10060.134303] ? hrtimer_interrupt+0x128/0x2c0 [10060.136330] ? smp_apic_timer_interrupt+0xa6/0x270 [10060.138656] ? apic_timer_interrupt+0xf/0x20 [10060.140602] [10060.141505] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10060.144322] do_raw_spin_lock+0x177/0x190 [10060.145907] _raw_spin_lock+0x21/0x30 [10060.147495] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10060.149905] ? do_raw_spin_unlock+0x75/0x190 [10060.151698] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10060.154047] ? do_wait_intr+0xf0/0xf0 [10060.155744] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10060.158325] mdc_close+0x53b/0xd70 [mdc] [10060.159918] lmv_close+0x18d/0x490 [lmv] [10060.161754] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10060.164671] ll_md_real_close+0xa5/0x2a0 [lustre] [10060.166765] ll_md_close+0x274/0x800 [lustre] [10060.168758] ll_file_release+0x128/0xc50 [lustre] [10060.170686] __fput+0x10d/0x440 [10060.171693] ____fput+0x12/0x20 [10060.172735] task_work_run+0xc6/0x110 [10060.173888] exit_to_usermode_loop+0x1dd/0x1f0 [10060.175982] do_syscall_64+0x3d6/0x440 [10060.177688] entry_SYSCALL_64_after_hwframe+0x49/0xae [10060.179754] RIP: 0033:0x7f1a63983cd8 [10060.181311] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10060.187115] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10060.189881] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10060.192653] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10060.195304] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10060.198024] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10060.200980] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10080.053028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10080.058131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10080.104094] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10080.112054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10080.116966] RIP: 0010:native_safe_halt+0x2e/0x50 [10080.124603] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10080.134821] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10080.143753] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10080.149950] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [10080.156183] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10080.161847] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10080.169932] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10080.174686] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10080.179185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10080.186593] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10080.193793] Call Trace: [10080.197693] [10080.199744] ? show_regs.cold.9+0x22/0x2f [10080.201761] ? watchdog_timer_fn.cold.6+0x62/0x10a [10080.204700] ? __hrtimer_run_queues+0x1c2/0x4d0 [10080.208605] ? watchdog+0x40/0x40 [10080.210602] ? hrtimer_interrupt+0x128/0x2c0 [10080.213676] ? smp_apic_timer_interrupt+0xa6/0x270 [10080.217132] ? apic_timer_interrupt+0xf/0x20 [10080.219902] [10080.220947] ? native_safe_halt+0x2e/0x50 [10080.223739] kvm_wait+0xcd/0xe0 [10080.225698] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10080.229089] do_raw_spin_lock+0x177/0x190 [10080.230970] _raw_spin_lock+0x21/0x30 [10080.232919] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10080.237510] ? do_raw_spin_unlock+0x75/0x190 [10080.238904] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10080.242066] ? do_wait_intr+0xf0/0xf0 [10080.244711] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10080.248740] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10080.251903] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10080.256099] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10080.263945] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10080.269406] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10080.271675] mdc_intent_lock+0x353/0x7e0 [mdc] [10080.274826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10080.279983] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10080.285260] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10080.287934] lmv_intent_open+0x338/0x1150 [lmv] [10080.291499] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10080.298468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10080.304936] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10080.307467] ll_intent_lock+0xc5/0xaa0 [lustre] [10080.310990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10080.316078] ll_lookup_it+0xe0e/0x2270 [lustre] [10080.319391] ? do_raw_spin_unlock+0x75/0x190 [10080.321801] ? _raw_spin_unlock+0x12/0x30 [10080.324564] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10080.326934] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10080.330705] ll_atomic_open+0x36a/0xd70 [lustre] [10080.334740] lookup_open+0x356/0xb40 [10080.337853] do_last+0x313/0xfc0 [10080.340254] ? nd_jump_root+0xe5/0x160 [10080.341405] ? path_init+0x437/0x520 [10080.342683] path_openat+0xf7/0x500 [10080.346473] ? do_raw_spin_unlock+0x75/0x190 [10080.350641] do_filp_open+0x99/0x140 [10080.356668] ? getname_flags+0x6e/0x330 [10080.358676] ? __check_object_size+0xff/0x256 [10080.360648] ? do_raw_spin_unlock+0x75/0x190 [10080.362545] ? _raw_spin_unlock+0x12/0x30 [10080.364406] do_sys_openat2+0x2b4/0x410 [10080.365857] do_sys_open+0x73/0xa0 [10080.367677] __x64_sys_openat+0x24/0x30 [10080.371033] do_syscall_64+0xc1/0x440 [10080.372086] entry_SYSCALL_64_after_hwframe+0x49/0xae [10080.375429] RIP: 0033:0x7f1a63983332 [10080.379478] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10080.395368] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10080.400504] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10080.407691] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10080.414650] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10080.421100] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10080.424777] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10084.056070] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10084.058182] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10084.058970] Modules linked in: [10084.062894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10084.062969] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10084.062976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10084.062979] RIP: 0010:native_safe_halt+0x2e/0x50 [10084.062992] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10084.062997] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10084.063010] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10084.063013] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10084.063015] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10084.063017] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10084.063204] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10084.066513] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10084.066579] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10084.066585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10084.066587] RIP: 0010:native_safe_halt+0x2e/0x50 [10084.066598] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10084.066602] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10084.066606] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10084.066608] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10084.066609] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10084.066611] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10084.066614] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10084.066616] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10084.066618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10084.066620] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10084.066628] Call Trace: [10084.066632] [10084.066637] ? show_regs.cold.9+0x22/0x2f [10084.066645] ? watchdog_timer_fn.cold.6+0x62/0x10a [10084.066651] ? __hrtimer_run_queues+0x1c2/0x4d0 [10084.066654] ? watchdog+0x40/0x40 [10084.066658] ? hrtimer_interrupt+0x128/0x2c0 [10084.066661] ? smp_apic_timer_interrupt+0xa6/0x270 [10084.066667] ? apic_timer_interrupt+0xf/0x20 [10084.066671] [10084.066673] ? native_safe_halt+0x2e/0x50 [10084.066676] kvm_wait+0xcd/0xe0 [10084.066682] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10084.066689] do_raw_spin_lock+0x177/0x190 [10084.066692] _raw_spin_lock+0x21/0x30 [10084.066695] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10084.066987] ? do_raw_spin_unlock+0x75/0x190 [10084.066993] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10084.067324] ? do_wait_intr+0xf0/0xf0 [10084.067332] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10084.067579] mdc_close+0x53b/0xd70 [mdc] [10084.067615] lmv_close+0x18d/0x490 [lmv] [10084.067638] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10084.067740] ll_md_real_close+0xa5/0x2a0 [lustre] [10084.067814] ll_md_close+0x274/0x800 [lustre] [10084.067889] ll_file_release+0x128/0xc50 [lustre] [10084.067964] __fput+0x10d/0x440 [10084.067972] ____fput+0x12/0x20 [10084.067975] task_work_run+0xc6/0x110 [10084.067981] exit_to_usermode_loop+0x1dd/0x1f0 [10084.067989] do_syscall_64+0x3d6/0x440 [10084.067994] entry_SYSCALL_64_after_hwframe+0x49/0xae [10084.068000] RIP: 0033:0x7f1a63983cd8 [10084.068006] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10084.068010] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10084.068015] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10084.068018] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10084.068020] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10084.068023] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10084.068026] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10084.311632] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10084.322203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10084.327780] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10084.332938] Call Trace: [10084.334517] [10084.335517] ? show_regs.cold.9+0x22/0x2f [10084.337106] ? watchdog_timer_fn.cold.6+0x62/0x10a [10084.339899] ? __hrtimer_run_queues+0x1c2/0x4d0 [10084.343613] ? watchdog+0x40/0x40 [10084.344879] ? hrtimer_interrupt+0x128/0x2c0 [10084.349755] ? smp_apic_timer_interrupt+0xa6/0x270 [10084.355232] ? apic_timer_interrupt+0xf/0x20 [10084.357725] [10084.359103] ? native_safe_halt+0x2e/0x50 [10084.364580] kvm_wait+0xcd/0xe0 [10084.368743] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10084.373854] do_raw_spin_lock+0x177/0x190 [10084.375785] _raw_spin_lock+0x21/0x30 [10084.377757] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10084.380699] obd_mod_free+0x46/0x140 [ptlrpc] [10084.383813] mdc_commit_open+0x91/0xb0 [mdc] [10084.385868] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10084.388884] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10084.391855] after_reply+0xe2c/0x1780 [ptlrpc] [10084.396902] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10084.402728] ? __next_timer_interrupt+0x160/0x160 [10084.406646] ptlrpcd+0xa82/0xe40 [ptlrpc] [10084.408762] ? do_wait_intr+0xf0/0xf0 [10084.410522] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10084.412903] kthread+0x1d1/0x200 [10084.414313] ? set_kthread_struct+0x70/0x70 [10084.415778] ret_from_fork+0x1f/0x30 [10085.857844] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8062s! [10085.861090] Showing busy workqueues and worker pools: [10085.863712] workqueue events: flags=0x0 [10085.872260] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10085.875731] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10085.879167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10085.883058] pending: sec_gc_main [ptlrpc] [10085.884924] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10085.889163] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10085.893642] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10085.897992] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10085.903636] workqueue events_long: flags=0x0 [10085.905662] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10085.908758] pending: lu_objects_flush [obdclass] [10085.912718] workqueue events_unbound: flags=0x2 [10085.916699] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10085.919790] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10085.924977] workqueue events_power_efficient: flags=0x80 [10085.926753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10085.933176] pending: neigh_periodic_work, do_cache_clean [10085.935779] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10085.939786] pending: neigh_periodic_work, check_lifetime [10085.942404] workqueue mm_percpu_wq: flags=0x8 [10085.944055] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10085.949246] pending: vmstat_update [10085.950732] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10085.955474] pending: vmstat_update [10085.957743] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10085.961243] pending: vmstat_update [10085.962912] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10085.969469] pending: vmstat_update [10085.970973] workqueue writeback: flags=0x4a [10085.975629] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10085.978723] pending: wb_workfn, wb_workfn [10085.980779] workqueue xprtiod: flags=0xa [10085.983235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10085.987357] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10085.993857] workqueue ipv6_addrconf: flags=0x40008 [10085.996563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10086.000918] pending: addrconf_verify_work [10086.003481] workqueue knbd0-recv: flags=0x1a [10086.005565] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10086.008507] in-flight: 647:recv_work [10086.011116] workqueue ptlrpc_pinger: flags=0x2 [10086.013550] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10086.018459] pending: ptlrpc_pinger_main [ptlrpc] [10086.021768] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10076s workers=2 idle: 652 [10088.057535] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10088.061289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10088.091804] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10088.097607] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10088.102455] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10088.105393] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10088.115652] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10088.119780] RAX: 00000000000044db RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10088.123423] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10088.127312] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10088.130756] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10088.134036] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10088.137782] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10088.142035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10088.144842] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10088.148328] Call Trace: [10088.149627] [10088.150706] ? show_regs.cold.9+0x22/0x2f [10088.152353] ? watchdog_timer_fn.cold.6+0x62/0x10a [10088.154833] ? __hrtimer_run_queues+0x1c2/0x4d0 [10088.156777] ? watchdog+0x40/0x40 [10088.158352] ? hrtimer_interrupt+0x128/0x2c0 [10088.160283] ? smp_apic_timer_interrupt+0xa6/0x270 [10088.162836] ? apic_timer_interrupt+0xf/0x20 [10088.164891] [10088.165824] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10088.168412] do_raw_spin_lock+0x177/0x190 [10088.170136] _raw_spin_lock+0x21/0x30 [10088.171880] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10088.174385] ? do_raw_spin_unlock+0x75/0x190 [10088.176506] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10088.179474] ? do_wait_intr+0xf0/0xf0 [10088.181326] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10088.183851] mdc_close+0x53b/0xd70 [mdc] [10088.185653] lmv_close+0x18d/0x490 [lmv] [10088.187635] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10088.189822] ll_md_real_close+0xa5/0x2a0 [lustre] [10088.192747] ll_md_close+0x274/0x800 [lustre] [10088.195096] ll_file_release+0x128/0xc50 [lustre] [10088.197811] __fput+0x10d/0x440 [10088.199397] ____fput+0x12/0x20 [10088.201285] task_work_run+0xc6/0x110 [10088.203064] exit_to_usermode_loop+0x1dd/0x1f0 [10088.205488] do_syscall_64+0x3d6/0x440 [10088.207155] entry_SYSCALL_64_after_hwframe+0x49/0xae [10088.210457] RIP: 0033:0x7f1a63983cd8 [10088.211745] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10088.219044] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10088.222058] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10088.224754] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10088.227515] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10088.231295] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10088.234397] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10112.058771] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10112.059028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10112.059537] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10112.059544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10112.059616] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10112.059622] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10112.059624] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10112.059638] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10112.059641] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10112.059645] RAX: 0000000000002f88 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10112.059648] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10112.059650] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10112.059652] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10112.059654] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10112.059657] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10112.059660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10112.059663] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10112.059671] Call Trace: [10112.059677] [10112.059683] ? show_regs.cold.9+0x22/0x2f [10112.059703] ? watchdog_timer_fn.cold.6+0x62/0x10a [10112.059710] ? __hrtimer_run_queues+0x1c2/0x4d0 [10112.059715] ? watchdog+0x40/0x40 [10112.059719] ? hrtimer_interrupt+0x128/0x2c0 [10112.059723] ? smp_apic_timer_interrupt+0xa6/0x270 [10112.059733] ? apic_timer_interrupt+0xf/0x20 [10112.059738] [10112.059741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10112.059747] do_raw_spin_lock+0x177/0x190 [10112.059752] _raw_spin_lock+0x21/0x30 [10112.059757] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10112.061207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10112.061275] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10112.061281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10112.061283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10112.061293] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10112.061297] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10112.061301] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10112.061303] RDX: 0000000000007520 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10112.061305] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10112.061307] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10112.061309] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10112.061312] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10112.061314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10112.061317] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10112.061324] Call Trace: [10112.061328] [10112.061332] ? show_regs.cold.9+0x22/0x2f [10112.061341] ? watchdog_timer_fn.cold.6+0x62/0x10a [10112.061347] ? __hrtimer_run_queues+0x1c2/0x4d0 [10112.061350] ? watchdog+0x40/0x40 [10112.061354] ? hrtimer_interrupt+0x128/0x2c0 [10112.061359] ? smp_apic_timer_interrupt+0xa6/0x270 [10112.061366] ? apic_timer_interrupt+0xf/0x20 [10112.061371] [10112.061374] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10112.061379] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10112.061384] do_raw_spin_lock+0x177/0x190 [10112.061389] _raw_spin_lock+0x21/0x30 [10112.061394] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10112.061808] ? do_raw_spin_unlock+0x75/0x190 [10112.061815] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10112.061677] obd_mod_free+0x46/0x140 [ptlrpc] [10112.063374] mdc_commit_open+0x91/0xb0 [mdc] [10112.063413] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10112.063671] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10112.063877] after_reply+0xe2c/0x1780 [ptlrpc] [10112.064369] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10112.064582] ? __next_timer_interrupt+0x160/0x160 [10112.064592] ptlrpcd+0xa82/0xe40 [ptlrpc] [10112.065178] ? do_wait_intr+0xf0/0xf0 [10112.065189] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10112.065400] kthread+0x1d1/0x200 [10112.065406] ? set_kthread_struct+0x70/0x70 [10112.065410] ret_from_fork+0x1f/0x30 [10112.072561] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10112.072826] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10112.072834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10112.072892] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10112.072907] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10112.072911] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10112.072916] RAX: 0000000000007a10 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10112.072919] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10112.072921] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10112.072923] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10112.072926] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10112.072929] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10112.072932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10112.072934] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10112.072953] Call Trace: [10112.072959] [10112.072965] ? show_regs.cold.9+0x22/0x2f [10112.072974] ? watchdog_timer_fn.cold.6+0x62/0x10a [10112.072981] ? __hrtimer_run_queues+0x1c2/0x4d0 [10112.072985] ? watchdog+0x40/0x40 [10112.072990] ? hrtimer_interrupt+0x128/0x2c0 [10112.072993] ? smp_apic_timer_interrupt+0xa6/0x270 [10112.073002] ? apic_timer_interrupt+0xf/0x20 [10112.073007] [10112.073010] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10112.073015] do_raw_spin_lock+0x177/0x190 [10112.073020] _raw_spin_lock+0x21/0x30 [10112.073024] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10112.085350] ? do_wait_intr+0xf0/0xf0 [10112.085371] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10112.148287] ? do_raw_spin_unlock+0x75/0x190 [10112.148309] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10112.149145] ? do_wait_intr+0xf0/0xf0 [10112.149156] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10112.150518] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10112.162156] mdc_close+0x53b/0xd70 [mdc] [10112.162207] lmv_close+0x18d/0x490 [lmv] [10112.162238] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10112.171075] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10112.171512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10112.171625] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10112.171673] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10112.171705] mdc_intent_lock+0x353/0x7e0 [mdc] [10112.171732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10112.172078] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10112.179036] ll_md_real_close+0xa5/0x2a0 [lustre] [10112.196188] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10112.196242] lmv_intent_open+0x338/0x1150 [lmv] [10112.196449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10112.196565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10112.196649] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10112.196674] ll_intent_lock+0xc5/0xaa0 [lustre] [10112.196758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10112.196952] ll_lookup_it+0xe0e/0x2270 [lustre] [10112.197033] ? do_raw_spin_unlock+0x75/0x190 [10112.197042] ? _raw_spin_unlock+0x12/0x30 [10112.197048] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10112.197074] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10112.197094] ll_atomic_open+0x36a/0xd70 [lustre] [10112.210609] ll_md_close+0x274/0x800 [lustre] [10112.211568] ll_file_release+0x128/0xc50 [lustre] [10112.211662] __fput+0x10d/0x440 [10112.211673] ____fput+0x12/0x20 [10112.211678] task_work_run+0xc6/0x110 [10112.211687] exit_to_usermode_loop+0x1dd/0x1f0 [10112.211705] do_syscall_64+0x3d6/0x440 [10112.211711] entry_SYSCALL_64_after_hwframe+0x49/0xae [10112.211721] RIP: 0033:0x7f1a63983cd8 [10112.211728] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10112.211732] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10112.211738] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10112.211740] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10112.211743] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10112.211745] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10112.211747] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10112.880524] lookup_open+0x356/0xb40 [10112.882671] do_last+0x313/0xfc0 [10112.884196] ? nd_jump_root+0xe5/0x160 [10112.888570] ? path_init+0x437/0x520 [10112.890621] path_openat+0xf7/0x500 [10112.893157] ? do_raw_spin_unlock+0x75/0x190 [10112.894438] do_filp_open+0x99/0x140 [10112.895500] ? getname_flags+0x6e/0x330 [10112.897747] ? __check_object_size+0xff/0x256 [10112.898883] ? do_raw_spin_unlock+0x75/0x190 [10112.905066] ? _raw_spin_unlock+0x12/0x30 [10112.906645] do_sys_openat2+0x2b4/0x410 [10112.908071] do_sys_open+0x73/0xa0 [10112.909092] __x64_sys_openat+0x24/0x30 [10112.910241] do_syscall_64+0xc1/0x440 [10112.911594] entry_SYSCALL_64_after_hwframe+0x49/0xae [10112.913530] RIP: 0033:0x7f1a63983332 [10112.915019] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10112.931682] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10112.936594] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10112.940057] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10112.945188] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10112.948246] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10112.952734] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10116.057043] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10116.064685] BUG: workqueue lockup - pool [10116.065761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10116.065975] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10116.065983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10116.065991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10116.066018] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10116.066023] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10116.066028] RAX: 0000000000004886 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10116.066040] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10116.066043] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10116.066045] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10116.066047] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10116.066051] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10116.066053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10116.066056] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10116.066063] Call Trace: [10116.066337] [10116.066345] ? show_regs.cold.9+0x22/0x2f [10116.066356] ? watchdog_timer_fn.cold.6+0x62/0x10a [10116.066362] ? __hrtimer_run_queues+0x1c2/0x4d0 [10116.066366] ? watchdog+0x40/0x40 [10116.066371] ? hrtimer_interrupt+0x128/0x2c0 [10116.066466] ? smp_apic_timer_interrupt+0xa6/0x270 [10116.066475] ? apic_timer_interrupt+0xf/0x20 [10116.066479] [10116.066482] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10116.066488] do_raw_spin_lock+0x177/0x190 [10116.066493] _raw_spin_lock+0x21/0x30 [10116.066498] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10116.066930] ? do_raw_spin_unlock+0x75/0x190 [10116.066937] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10116.067285] ? do_wait_intr+0xf0/0xf0 [10116.067294] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10116.067640] mdc_close+0x53b/0xd70 [mdc] [10116.067693] lmv_close+0x18d/0x490 [lmv] [10116.067722] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10116.067894] ll_md_real_close+0xa5/0x2a0 [lustre] [10116.067983] ll_md_close+0x274/0x800 [lustre] [10116.068077] ll_file_release+0x128/0xc50 [lustre] [10116.068168] __fput+0x10d/0x440 [10116.068176] ____fput+0x12/0x20 [10116.068180] task_work_run+0xc6/0x110 [10116.068187] exit_to_usermode_loop+0x1dd/0x1f0 [10116.068193] do_syscall_64+0x3d6/0x440 [10116.068197] entry_SYSCALL_64_after_hwframe+0x49/0xae [10116.069108] RIP: 0033:0x7f1a63983cd8 [10116.072834] cpus=0-3 flags=0x4 nice=0 stuck for 8093s! [10116.072857] Showing busy workqueues and worker pools: [10116.072861] workqueue events: flags=0x0 [10116.096101] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10116.096115] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10116.096127] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10116.096129] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10116.096131] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10116.096133] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10116.096135] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10116.356053] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10116.363928] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10116.375325] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10116.382719] pending: sec_gc_main [ptlrpc] [10116.387820] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10116.393367] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10116.401992] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10116.409383] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10116.419188] workqueue events_long: flags=0x0 [10116.423377] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10116.434443] pending: lu_objects_flush [obdclass] [10116.440486] workqueue events_unbound: flags=0x2 [10116.445204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10116.452196] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10116.466052] workqueue events_power_efficient: flags=0x80 [10116.471428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10116.478241] pending: neigh_periodic_work, do_cache_clean [10116.481481] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10116.491411] pending: neigh_periodic_work, check_lifetime [10116.499390] workqueue mm_percpu_wq: flags=0x8 [10116.506765] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10116.517014] pending: vmstat_update [10116.521082] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10116.532839] pending: vmstat_update [10116.535332] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10116.542127] pending: vmstat_update [10116.544929] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10116.550256] pending: vmstat_update [10116.556837] workqueue writeback: flags=0x4a [10116.559821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10116.571916] pending: wb_workfn, wb_workfn [10116.575652] workqueue xprtiod: flags=0xa [10116.577117] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10116.581451] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10116.586977] workqueue ipv6_addrconf: flags=0x40008 [10116.592025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10116.599273] pending: addrconf_verify_work [10116.601167] workqueue knbd0-recv: flags=0x1a [10116.605063] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10116.610504] in-flight: 647:recv_work [10116.612233] workqueue ptlrpc_pinger: flags=0x2 [10116.616384] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10116.619348] pending: ptlrpc_pinger_main [ptlrpc] [10116.622685] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10106s workers=2 idle: 652 [10140.056021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10140.058274] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10140.059546] Modules linked in: lustre(O) osp(O) [10140.064996] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10140.065152] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10140.065159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10140.065162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10140.065176] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10140.065180] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10140.065918] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10140.065925] RDX: 0000000000002eca RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10140.065931] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10140.065934] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10140.065939] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10140.065942] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10140.065946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10140.065948] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10140.065957] Call Trace: [10140.065965] [10140.065972] ? show_regs.cold.9+0x22/0x2f [10140.065987] ? watchdog_timer_fn.cold.6+0x62/0x10a [10140.065996] ? __hrtimer_run_queues+0x1c2/0x4d0 [10140.066000] ? watchdog+0x40/0x40 [10140.066005] ? hrtimer_interrupt+0x128/0x2c0 [10140.066009] ? smp_apic_timer_interrupt+0xa6/0x270 [10140.066018] ? apic_timer_interrupt+0xf/0x20 [10140.066022] [10140.066025] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10140.066031] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10140.066035] do_raw_spin_lock+0x177/0x190 [10140.066040] _raw_spin_lock+0x21/0x30 [10140.066047] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10140.068936] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10140.069194] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10140.069203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10140.069206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10140.069222] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10140.069227] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10140.069231] RAX: 0000000000003fee RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10140.069233] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10140.069235] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10140.069237] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10140.069239] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10140.069243] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10140.069246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10140.069248] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10140.069257] Call Trace: [10140.069263] [10140.069269] ? show_regs.cold.9+0x22/0x2f [10140.069279] ? watchdog_timer_fn.cold.6+0x62/0x10a [10140.069285] ? __hrtimer_run_queues+0x1c2/0x4d0 [10140.069289] ? watchdog+0x40/0x40 [10140.069293] ? hrtimer_interrupt+0x128/0x2c0 [10140.069297] ? smp_apic_timer_interrupt+0xa6/0x270 [10140.069306] ? apic_timer_interrupt+0xf/0x20 [10140.069311] [10140.069313] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10140.069319] do_raw_spin_lock+0x177/0x190 [10140.069324] _raw_spin_lock+0x21/0x30 [10140.069328] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10140.069703] ? do_raw_spin_unlock+0x75/0x190 [10140.069710] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10140.070083] ? do_wait_intr+0xf0/0xf0 [10140.070092] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10140.070410] mdc_close+0x53b/0xd70 [mdc] [10140.070456] lmv_close+0x18d/0x490 [lmv] [10140.070484] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10140.070631] ll_md_real_close+0xa5/0x2a0 [lustre] [10140.071014] ll_md_close+0x274/0x800 [lustre] [10140.071100] ll_file_release+0x128/0xc50 [lustre] [10140.071194] __fput+0x10d/0x440 [10140.071204] ____fput+0x12/0x20 [10140.071209] task_work_run+0xc6/0x110 [10140.071218] exit_to_usermode_loop+0x1dd/0x1f0 [10140.071226] do_syscall_64+0x3d6/0x440 [10140.071232] entry_SYSCALL_64_after_hwframe+0x49/0xae [10140.071244] RIP: 0033:0x7f1a63983cd8 [10140.071255] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10140.071259] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10140.071264] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10140.071266] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10140.071268] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10140.071271] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10140.071273] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10140.328057] obd_mod_free+0x46/0x140 [ptlrpc] [10140.330281] mdc_commit_open+0x91/0xb0 [mdc] [10140.332198] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10140.335052] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10140.337676] after_reply+0xe2c/0x1780 [ptlrpc] [10140.339949] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10140.342536] ? __next_timer_interrupt+0x160/0x160 [10140.344745] ptlrpcd+0xa82/0xe40 [ptlrpc] [10140.346759] ? do_wait_intr+0xf0/0xf0 [10140.348993] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10140.351548] kthread+0x1d1/0x200 [10140.353374] ? set_kthread_struct+0x70/0x70 [10140.355951] ret_from_fork+0x1f/0x30 [10144.053298] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10144.057628] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10144.065793] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10144.065871] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10144.065878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10144.065885] RIP: 0010:native_safe_halt+0x2e/0x50 [10144.065903] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10144.065913] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10144.065919] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10144.065921] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [10144.065924] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10144.065926] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10144.065932] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10144.065936] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10144.065939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10144.065942] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10144.065950] Call Trace: [10144.065957] [10144.065967] ? show_regs.cold.9+0x22/0x2f [10144.065985] ? watchdog_timer_fn.cold.6+0x62/0x10a [10144.065993] ? __hrtimer_run_queues+0x1c2/0x4d0 [10144.066001] ? watchdog+0x40/0x40 [10144.066005] ? hrtimer_interrupt+0x128/0x2c0 [10144.066010] ? smp_apic_timer_interrupt+0xa6/0x270 [10144.066021] ? apic_timer_interrupt+0xf/0x20 [10144.066025] [10144.066029] ? native_safe_halt+0x2e/0x50 [10144.066033] kvm_wait+0xcd/0xe0 [10144.066042] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10144.066052] do_raw_spin_lock+0x177/0x190 [10144.066057] _raw_spin_lock+0x21/0x30 [10144.066061] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10144.066480] ? do_raw_spin_unlock+0x75/0x190 [10144.066486] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10144.066732] ? do_wait_intr+0xf0/0xf0 [10144.066745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10144.067168] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10144.067436] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10144.067798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10144.067933] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10144.067984] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10144.068019] mdc_intent_lock+0x353/0x7e0 [mdc] [10144.068047] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10144.068149] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10144.068397] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10144.068660] lmv_intent_open+0x338/0x1150 [lmv] [10144.068703] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10144.068806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10144.068886] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10144.068909] ll_intent_lock+0xc5/0xaa0 [lustre] [10144.069001] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10144.069088] ll_lookup_it+0xe0e/0x2270 [lustre] [10144.080556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10144.080660] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10144.080669] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10144.080672] RIP: 0010:native_safe_halt+0x2e/0x50 [10144.080689] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10144.080694] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10144.080702] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10144.080704] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [10144.080706] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10144.080709] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10144.080711] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10144.080714] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10144.080717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10144.080720] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10144.080727] Call Trace: [10144.080736] [10144.080745] ? show_regs.cold.9+0x22/0x2f [10144.080758] ? watchdog_timer_fn.cold.6+0x62/0x10a [10144.080764] ? __hrtimer_run_queues+0x1c2/0x4d0 [10144.080769] ? watchdog+0x40/0x40 [10144.080773] ? hrtimer_interrupt+0x128/0x2c0 [10144.080776] ? smp_apic_timer_interrupt+0xa6/0x270 [10144.080783] ? apic_timer_interrupt+0xf/0x20 [10144.080787] [10144.080790] ? native_safe_halt+0x2e/0x50 [10144.080794] kvm_wait+0xcd/0xe0 [10144.080802] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10144.080811] do_raw_spin_lock+0x177/0x190 [10144.080816] _raw_spin_lock+0x21/0x30 [10144.080820] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10144.081366] ? do_raw_spin_unlock+0x75/0x190 [10144.081372] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10144.081749] ? do_wait_intr+0xf0/0xf0 [10144.081758] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10144.081969] mdc_close+0x53b/0xd70 [mdc] [10144.082011] lmv_close+0x18d/0x490 [lmv] [10144.082051] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10144.082188] ll_md_real_close+0xa5/0x2a0 [lustre] [10144.082398] ll_md_close+0x274/0x800 [lustre] [10144.082625] ll_file_release+0x128/0xc50 [lustre] [10144.082723] __fput+0x10d/0x440 [10144.082736] ____fput+0x12/0x20 [10144.082741] task_work_run+0xc6/0x110 [10144.082751] exit_to_usermode_loop+0x1dd/0x1f0 [10144.082759] do_syscall_64+0x3d6/0x440 [10144.113498] ? do_raw_spin_unlock+0x75/0x190 [10144.113521] ? _raw_spin_unlock+0x12/0x30 [10144.113530] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10144.113693] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10144.113714] ll_atomic_open+0x36a/0xd70 [lustre] [10144.113885] lookup_open+0x356/0xb40 [10144.113896] do_last+0x313/0xfc0 [10144.113902] ? nd_jump_root+0xe5/0x160 [10144.113907] ? path_init+0x437/0x520 [10144.113911] path_openat+0xf7/0x500 [10144.113916] ? do_raw_spin_unlock+0x75/0x190 [10144.113921] do_filp_open+0x99/0x140 [10144.113928] ? getname_flags+0x6e/0x330 [10144.113933] ? __check_object_size+0xff/0x256 [10144.113942] ? do_raw_spin_unlock+0x75/0x190 [10144.113947] ? _raw_spin_unlock+0x12/0x30 [10144.113952] do_sys_openat2+0x2b4/0x410 [10144.113956] do_sys_open+0x73/0xa0 [10144.113961] __x64_sys_openat+0x24/0x30 [10144.113964] do_syscall_64+0xc1/0x440 [10144.113973] entry_SYSCALL_64_after_hwframe+0x49/0xae [10144.113981] RIP: 0033:0x7f1a63983332 [10144.113992] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10144.113996] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10144.114002] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10144.114004] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10144.114007] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10144.114009] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10144.114011] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10144.535824] entry_SYSCALL_64_after_hwframe+0x49/0xae [10144.538160] RIP: 0033:0x7f1a63983cd8 [10144.540029] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10144.548955] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10144.553133] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10144.556569] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10144.560100] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10144.563849] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10144.567411] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10146.784872] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8123s! [10146.790079] Showing busy workqueues and worker pools: [10146.793764] workqueue events: flags=0x0 [10146.796487] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10146.800000] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10146.802751] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10146.805484] pending: sec_gc_main [ptlrpc] [10146.808324] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10146.811480] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10146.816505] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10146.819367] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10146.823685] workqueue events_long: flags=0x0 [10146.826162] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10146.829037] pending: lu_objects_flush [obdclass] [10146.831502] workqueue events_unbound: flags=0x2 [10146.832988] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10146.835397] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10146.839357] workqueue events_power_efficient: flags=0x80 [10146.843314] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10146.845978] pending: neigh_periodic_work, do_cache_clean [10146.848149] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10146.851657] pending: neigh_periodic_work, check_lifetime [10146.854101] workqueue mm_percpu_wq: flags=0x8 [10146.855738] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10146.859420] pending: vmstat_update [10146.861034] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10146.864486] pending: vmstat_update [10146.866411] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10146.869408] pending: vmstat_update [10146.870976] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10146.873796] pending: vmstat_update [10146.875449] workqueue writeback: flags=0x4a [10146.877170] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10146.880053] pending: wb_workfn, wb_workfn [10146.882074] workqueue xprtiod: flags=0xa [10146.883528] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10146.886079] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10146.889861] workqueue ipv6_addrconf: flags=0x40008 [10146.891645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10146.894478] pending: addrconf_verify_work [10146.896310] workqueue knbd0-recv: flags=0x1a [10146.898083] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10146.900831] in-flight: 647:recv_work [10146.902719] workqueue ptlrpc_pinger: flags=0x2 [10146.904853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10146.907195] pending: ptlrpc_pinger_main [ptlrpc] [10146.909496] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10136s workers=2 idle: 652 [10168.056433] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10168.058269] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10168.067068] Modules linked in: [10168.075531] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10168.075615] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10168.075624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10168.075627] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10168.075649] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10168.075653] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10168.075657] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10168.075660] RDX: 0000000000001cbc RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10168.075663] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10168.075665] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10168.075667] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10168.075670] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10168.075673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10168.075688] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10168.075695] Call Trace: [10168.075700] [10168.075706] ? show_regs.cold.9+0x22/0x2f [10168.075715] ? watchdog_timer_fn.cold.6+0x62/0x10a [10168.075728] ? __hrtimer_run_queues+0x1c2/0x4d0 [10168.075732] ? watchdog+0x40/0x40 [10168.075736] ? hrtimer_interrupt+0x128/0x2c0 [10168.075741] ? smp_apic_timer_interrupt+0xa6/0x270 [10168.075749] ? apic_timer_interrupt+0xf/0x20 [10168.075754] [10168.075756] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10168.075761] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10168.075765] do_raw_spin_lock+0x177/0x190 [10168.075770] _raw_spin_lock+0x21/0x30 [10168.075780] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10168.076153] obd_mod_free+0x46/0x140 [ptlrpc] [10168.076784] mdc_commit_open+0x91/0xb0 [mdc] [10168.076826] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10168.077062] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10168.077474] after_reply+0xe2c/0x1780 [ptlrpc] [10168.077662] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10168.079000] ? __next_timer_interrupt+0x160/0x160 [10168.079011] ptlrpcd+0xa82/0xe40 [ptlrpc] [10168.079419] ? do_wait_intr+0xf0/0xf0 [10168.079429] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10168.081398] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10168.081493] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10168.081502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10168.081505] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10168.081521] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10168.081526] RSP: 0018:ffffc38542073af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10168.081531] RAX: 0000000000005276 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10168.081534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10168.081536] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10168.081539] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10168.081541] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10168.081545] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10168.081548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10168.081551] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10168.081561] Call Trace: [10168.081567] [10168.081574] ? show_regs.cold.9+0x22/0x2f [10168.081587] ? watchdog_timer_fn.cold.6+0x62/0x10a [10168.081594] ? __hrtimer_run_queues+0x1c2/0x4d0 [10168.081599] ? watchdog+0x40/0x40 [10168.081604] ? hrtimer_interrupt+0x128/0x2c0 [10168.081608] ? smp_apic_timer_interrupt+0xa6/0x270 [10168.081615] ? apic_timer_interrupt+0xf/0x20 [10168.081619] [10168.081622] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10168.081629] do_raw_spin_lock+0x177/0x190 [10168.081634] _raw_spin_lock+0x21/0x30 [10168.081639] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10168.082105] ? do_raw_spin_unlock+0x75/0x190 [10168.082112] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10168.082587] ? do_wait_intr+0xf0/0xf0 [10168.082596] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10168.082897] mdc_close+0x53b/0xd70 [mdc] [10168.082952] lmv_close+0x18d/0x490 [lmv] [10168.082995] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10168.083135] ll_md_real_close+0xa5/0x2a0 [lustre] [10168.083786] ll_md_close+0x274/0x800 [lustre] [10168.083941] ll_file_release+0x128/0xc50 [lustre] [10168.084025] __fput+0x10d/0x440 [10168.084036] ____fput+0x12/0x20 [10168.084040] task_work_run+0xc6/0x110 [10168.084047] exit_to_usermode_loop+0x1dd/0x1f0 [10168.084055] do_syscall_64+0x3d6/0x440 [10168.084059] entry_SYSCALL_64_after_hwframe+0x49/0xae [10168.084068] RIP: 0033:0x7f1a63983cd8 [10168.084076] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10168.084080] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10168.084084] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10168.084086] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10168.084088] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10168.084091] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10168.084093] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10168.624729] kthread+0x1d1/0x200 [10168.625879] ? set_kthread_struct+0x70/0x70 [10168.627803] ret_from_fork+0x1f/0x30 [10172.053036] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10172.055881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10172.057003] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10172.117709] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10172.117722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10172.117724] RIP: 0010:native_safe_halt+0x2e/0x50 [10172.117736] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10172.117740] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10172.117744] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10172.117746] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [10172.117747] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10172.117749] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10172.117751] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10172.117755] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10172.117758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10172.117760] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10172.117768] Call Trace: [10172.117774] [10172.117780] ? show_regs.cold.9+0x22/0x2f [10172.117790] ? watchdog_timer_fn.cold.6+0x62/0x10a [10172.117797] ? __hrtimer_run_queues+0x1c2/0x4d0 [10172.117802] ? watchdog+0x40/0x40 [10172.117806] ? hrtimer_interrupt+0x128/0x2c0 [10172.117811] ? smp_apic_timer_interrupt+0xa6/0x270 [10172.117818] ? apic_timer_interrupt+0xf/0x20 [10172.117822] [10172.117824] ? native_safe_halt+0x2e/0x50 [10172.117827] kvm_wait+0xcd/0xe0 [10172.117835] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10172.117843] do_raw_spin_lock+0x177/0x190 [10172.117848] _raw_spin_lock+0x21/0x30 [10172.117852] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10172.120078] ? do_raw_spin_unlock+0x75/0x190 [10172.120088] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10172.121720] ? do_wait_intr+0xf0/0xf0 [10172.121733] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10172.123641] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10172.123872] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10172.127015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10172.127127] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10172.127167] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10172.127205] mdc_intent_lock+0x353/0x7e0 [mdc] [10172.127231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10172.127331] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10172.128435] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10172.128474] lmv_intent_open+0x338/0x1150 [lmv] [10172.128506] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10172.128604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10172.128686] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10172.128708] ll_intent_lock+0xc5/0xaa0 [lustre] [10172.128790] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10172.128873] ll_lookup_it+0xe0e/0x2270 [lustre] [10172.128969] ? do_raw_spin_unlock+0x75/0x190 [10172.128977] ? _raw_spin_unlock+0x12/0x30 [10172.128983] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10172.129008] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10172.129029] ll_atomic_open+0x36a/0xd70 [lustre] [10172.129112] lookup_open+0x356/0xb40 [10172.129123] do_last+0x313/0xfc0 [10172.129128] ? nd_jump_root+0xe5/0x160 [10172.129133] ? path_init+0x437/0x520 [10172.129138] path_openat+0xf7/0x500 [10172.129143] ? do_raw_spin_unlock+0x75/0x190 [10172.129148] do_filp_open+0x99/0x140 [10172.129154] ? getname_flags+0x6e/0x330 [10172.129159] ? __check_object_size+0xff/0x256 [10172.129166] ? do_raw_spin_unlock+0x75/0x190 [10172.129169] ? _raw_spin_unlock+0x12/0x30 [10172.129174] do_sys_openat2+0x2b4/0x410 [10172.129178] do_sys_open+0x73/0xa0 [10172.129181] __x64_sys_openat+0x24/0x30 [10172.129185] do_syscall_64+0xc1/0x440 [10172.129190] entry_SYSCALL_64_after_hwframe+0x49/0xae [10172.129197] RIP: 0033:0x7f1a63983332 [10172.129205] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10172.129209] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10172.129214] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10172.129217] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10172.129219] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10172.129221] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10172.129223] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10172.892470] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10172.963032] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10172.990671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10173.016438] RIP: 0010:native_safe_halt+0x2e/0x50 [10173.022597] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10173.058883] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10173.071940] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10173.084452] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [10173.086988] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10173.108804] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10173.121841] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10173.131877] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10173.147516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10173.157917] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10173.175589] Call Trace: [10173.179064] [10173.180859] ? show_regs.cold.9+0x22/0x2f [10173.189428] ? watchdog_timer_fn.cold.6+0x62/0x10a [10173.192312] ? __hrtimer_run_queues+0x1c2/0x4d0 [10173.203903] ? watchdog+0x40/0x40 [10173.215428] ? hrtimer_interrupt+0x128/0x2c0 [10173.227532] ? smp_apic_timer_interrupt+0xa6/0x270 [10173.238708] ? apic_timer_interrupt+0xf/0x20 [10173.245953] [10173.249943] ? native_safe_halt+0x2e/0x50 [10173.259202] kvm_wait+0xcd/0xe0 [10173.263056] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10173.276897] do_raw_spin_lock+0x177/0x190 [10173.297965] _raw_spin_lock+0x21/0x30 [10173.299709] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10173.315111] ? do_raw_spin_unlock+0x75/0x190 [10173.327218] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10173.342681] ? do_wait_intr+0xf0/0xf0 [10173.347633] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10173.362435] mdc_close+0x53b/0xd70 [mdc] [10173.372753] lmv_close+0x18d/0x490 [lmv] [10173.379555] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10173.388725] ll_md_real_close+0xa5/0x2a0 [lustre] [10173.399222] ll_md_close+0x274/0x800 [lustre] [10173.405754] ll_file_release+0x128/0xc50 [lustre] [10173.417098] __fput+0x10d/0x440 [10173.418742] ____fput+0x12/0x20 [10173.423465] task_work_run+0xc6/0x110 [10173.433475] exit_to_usermode_loop+0x1dd/0x1f0 [10173.435066] do_syscall_64+0x3d6/0x440 [10173.443584] entry_SYSCALL_64_after_hwframe+0x49/0xae [10173.456827] RIP: 0033:0x7f1a63983cd8 [10173.459829] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10173.493195] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10173.500183] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10173.510738] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10173.517955] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10173.526117] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10173.545789] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10176.997505] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8153s! [10177.011928] Showing busy workqueues and worker pools: [10177.027953] workqueue events: flags=0x0 [10177.035703] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10177.050824] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10177.067938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10177.073853] pending: sec_gc_main [ptlrpc] [10177.085000] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10177.101677] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10177.125914] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10177.127753] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10177.154644] workqueue events_long: flags=0x0 [10177.168057] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10177.185334] pending: lu_objects_flush [obdclass] [10177.198504] workqueue events_unbound: flags=0x2 [10177.199936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10177.213513] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10177.242576] workqueue events_power_efficient: flags=0x80 [10177.244647] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10177.263864] pending: neigh_periodic_work, do_cache_clean [10177.283907] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10177.299507] pending: neigh_periodic_work, check_lifetime [10177.304953] workqueue mm_percpu_wq: flags=0x8 [10177.317190] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10177.331804] pending: vmstat_update [10177.335936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10177.350939] pending: vmstat_update [10177.364358] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10177.366882] pending: vmstat_update [10177.380474] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10177.396399] pending: vmstat_update [10177.400602] workqueue writeback: flags=0x4a [10177.409896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10177.411844] pending: wb_workfn, wb_workfn [10177.429683] workqueue xprtiod: flags=0xa [10177.431125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10177.445570] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10177.459242] workqueue ipv6_addrconf: flags=0x40008 [10177.461070] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10177.476509] pending: addrconf_verify_work [10177.485305] workqueue knbd0-recv: flags=0x1a [10177.486832] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10177.501166] in-flight: 647:recv_work [10177.519671] workqueue ptlrpc_pinger: flags=0x2 [10177.524672] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10177.541836] pending: ptlrpc_pinger_main [ptlrpc] [10177.550975] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10167s workers=2 idle: 652 [10182.631219] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [10182.639791] rcu: 3-...!: (5155629 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [10182.660493] (detected by 1, t=8160225 jiffies, g=269633, q=2912) [10182.681070] Sending NMI from CPU 1 to CPUs 3: [10182.701669] NMI backtrace for cpu 3 [10182.701677] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10182.701679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10182.701680] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10182.701682] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10182.701684] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [10182.701686] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10182.701688] RDX: 00000000000049eb RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10182.701690] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10182.701691] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10182.701692] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10182.701693] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10182.701695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10182.701696] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10182.701697] Call Trace: [10182.701698] [10182.701699] ? show_regs.cold.9+0x22/0x2f [10182.701700] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10182.701701] ? nmi_cpu_backtrace_handler+0x15/0x40 [10182.701702] ? nmi_handle+0xbd/0x210 [10182.701703] ? default_do_nmi+0x4b/0x1b0 [10182.701704] ? do_nmi+0x280/0x380 [10182.701705] ? end_repeat_nmi+0x16/0x1f [10182.701706] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10182.701708] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10182.701709] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10182.701710] [10182.701711] do_raw_spin_lock+0x177/0x190 [10182.701712] _raw_spin_lock+0x21/0x30 [10182.701713] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10182.701714] obd_mod_free+0x46/0x140 [ptlrpc] [10182.701715] mdc_commit_open+0x91/0xb0 [mdc] [10182.701716] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10182.701717] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10182.701718] after_reply+0xe2c/0x1780 [ptlrpc] [10182.701719] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10182.701720] ? __next_timer_interrupt+0x160/0x160 [10182.701721] ptlrpcd+0xa82/0xe40 [ptlrpc] [10182.701723] ? do_wait_intr+0xf0/0xf0 [10182.701724] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10182.701725] kthread+0x1d1/0x200 [10182.701726] ? set_kthread_struct+0x70/0x70 [10182.701727] ret_from_fork+0x1f/0x30 [10182.704623] rcu: rcu_sched kthread starved for 8159871 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [10183.205462] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10183.222822] rcu: RCU grace-period kthread stack dump: [10183.229521] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10183.246618] Call Trace: [10183.253380] __schedule+0x351/0xcb0 [10183.261086] schedule+0xc0/0x180 [10183.271552] schedule_timeout+0xb4/0x190 [10183.273564] ? do_raw_spin_unlock+0x75/0x190 [10183.277541] ? __next_timer_interrupt+0x160/0x160 [10183.291224] ? prepare_to_swait_event+0xf3/0x140 [10183.299685] rcu_gp_kthread+0x8dd/0x15b0 [10183.314584] ? force_qs_rnp+0x2c0/0x2c0 [10183.322535] kthread+0x1d1/0x200 [10183.330545] ? set_kthread_struct+0x70/0x70 [10183.332197] ret_from_fork+0x1f/0x30 [10183.333622] rcu: Stack dump where RCU GP kthread last ran: [10183.351357] Sending NMI from CPU 1 to CPUs 3: [10183.360755] NMI backtrace for cpu 3 [10183.360763] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10183.360765] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10183.360767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10183.360769] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10183.360771] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [10183.360773] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10183.360775] RDX: 0000000000007f10 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10183.360776] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10183.360778] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10183.360779] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10183.360781] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10183.360782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10183.360784] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10183.360785] Call Trace: [10183.360786] [10183.360787] ? show_regs.cold.9+0x22/0x2f [10183.360788] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10183.360789] ? nmi_cpu_backtrace_handler+0x15/0x40 [10183.360790] ? nmi_handle+0xbd/0x210 [10183.360791] ? default_do_nmi+0x4b/0x1b0 [10183.360792] ? do_nmi+0x280/0x380 [10183.360793] ? end_repeat_nmi+0x16/0x1f [10183.360794] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10183.360796] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10183.360797] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10183.360798] [10183.360799] do_raw_spin_lock+0x177/0x190 [10183.360800] _raw_spin_lock+0x21/0x30 [10183.360801] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10183.360802] obd_mod_free+0x46/0x140 [ptlrpc] [10183.360804] mdc_commit_open+0x91/0xb0 [mdc] [10183.360805] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10183.360806] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10183.360807] after_reply+0xe2c/0x1780 [ptlrpc] [10183.360809] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10183.360810] ? __next_timer_interrupt+0x160/0x160 [10183.360811] ptlrpcd+0xa82/0xe40 [ptlrpc] [10183.360812] ? do_wait_intr+0xf0/0xf0 [10183.360814] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10183.360815] kthread+0x1d1/0x200 [10183.360816] ? set_kthread_struct+0x70/0x70 [10183.360817] ret_from_fork+0x1f/0x30 [10200.059753] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10200.088170] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10200.185066] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10200.204255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10200.225297] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10200.234825] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10200.275000] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10200.298197] RAX: 00000000000017ee RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10200.310713] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10200.315574] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10200.321920] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10200.337248] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10200.346688] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10200.366408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10200.374016] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10200.383699] Call Trace: [10200.387961] [10200.390853] ? show_regs.cold.9+0x22/0x2f [10200.396929] ? watchdog_timer_fn.cold.6+0x62/0x10a [10200.407410] ? __hrtimer_run_queues+0x1c2/0x4d0 [10200.425867] ? watchdog+0x40/0x40 [10200.433179] ? hrtimer_interrupt+0x128/0x2c0 [10200.434886] ? smp_apic_timer_interrupt+0xa6/0x270 [10200.452558] ? apic_timer_interrupt+0xf/0x20 [10200.458366] [10200.460555] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10200.466410] do_raw_spin_lock+0x177/0x190 [10200.467945] _raw_spin_lock+0x21/0x30 [10200.472706] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10200.481906] ? do_raw_spin_unlock+0x75/0x190 [10200.486644] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10200.490951] ? do_wait_intr+0xf0/0xf0 [10200.499317] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10200.502805] mdc_close+0x53b/0xd70 [mdc] [10200.511582] lmv_close+0x18d/0x490 [lmv] [10200.518749] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10200.535674] ll_md_real_close+0xa5/0x2a0 [lustre] [10200.537558] ll_md_close+0x274/0x800 [lustre] [10200.553490] ll_file_release+0x128/0xc50 [lustre] [10200.555931] __fput+0x10d/0x440 [10200.565844] ____fput+0x12/0x20 [10200.569801] task_work_run+0xc6/0x110 [10200.572846] exit_to_usermode_loop+0x1dd/0x1f0 [10200.584833] do_syscall_64+0x3d6/0x440 [10200.593928] entry_SYSCALL_64_after_hwframe+0x49/0xae [10200.603589] RIP: 0033:0x7f1a63983cd8 [10200.609732] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10200.637743] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10200.649913] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10200.663791] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10200.674600] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10200.688282] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10200.701130] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10204.053589] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [10204.057942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10204.132850] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10204.153844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10204.169461] RIP: 0010:native_safe_halt+0x2e/0x50 [10204.175683] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10204.216301] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10204.231784] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10204.246039] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10204.259729] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10204.275325] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10204.285487] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10204.297764] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10204.311746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10204.325583] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10204.333663] Call Trace: [10204.336339] [10204.338380] ? show_regs.cold.9+0x22/0x2f [10204.343712] ? watchdog_timer_fn.cold.6+0x62/0x10a [10204.355570] ? __hrtimer_run_queues+0x1c2/0x4d0 [10204.369418] ? watchdog+0x40/0x40 [10204.371504] ? hrtimer_interrupt+0x128/0x2c0 [10204.388223] ? smp_apic_timer_interrupt+0xa6/0x270 [10204.409974] ? apic_timer_interrupt+0xf/0x20 [10204.411782] [10204.421553] ? native_safe_halt+0x2e/0x50 [10204.432081] kvm_wait+0xcd/0xe0 [10204.444563] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10204.457763] do_raw_spin_lock+0x177/0x190 [10204.473499] _raw_spin_lock+0x21/0x30 [10204.487472] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10204.493350] ? do_raw_spin_unlock+0x75/0x190 [10204.502277] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10204.513609] ? do_wait_intr+0xf0/0xf0 [10204.524781] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10204.539356] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10204.556970] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10204.580522] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10204.592717] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10204.612873] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10204.633964] mdc_intent_lock+0x353/0x7e0 [mdc] [10204.640712] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10204.648684] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10204.653440] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10204.657052] lmv_intent_open+0x338/0x1150 [lmv] [10204.662430] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10204.665538] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10204.673326] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10204.674978] ll_intent_lock+0xc5/0xaa0 [lustre] [10204.681597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10204.689790] ll_lookup_it+0xe0e/0x2270 [lustre] [10204.699038] ? do_raw_spin_unlock+0x75/0x190 [10204.700619] ? _raw_spin_unlock+0x12/0x30 [10204.714718] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10204.726863] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10204.736136] ll_atomic_open+0x36a/0xd70 [lustre] [10204.741521] lookup_open+0x356/0xb40 [10204.746468] do_last+0x313/0xfc0 [10204.754896] ? nd_jump_root+0xe5/0x160 [10204.761526] ? path_init+0x437/0x520 [10204.774355] path_openat+0xf7/0x500 [10204.781436] ? do_raw_spin_unlock+0x75/0x190 [10204.787752] do_filp_open+0x99/0x140 [10204.794665] ? getname_flags+0x6e/0x330 [10204.802557] ? __check_object_size+0xff/0x256 [10204.807209] ? do_raw_spin_unlock+0x75/0x190 [10204.810443] ? _raw_spin_unlock+0x12/0x30 [10204.813256] do_sys_openat2+0x2b4/0x410 [10204.814974] do_sys_open+0x73/0xa0 [10204.818059] __x64_sys_openat+0x24/0x30 [10204.823325] do_syscall_64+0xc1/0x440 [10204.826526] entry_SYSCALL_64_after_hwframe+0x49/0xae [10204.830288] RIP: 0033:0x7f1a63983332 [10204.837762] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10204.856713] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10204.862614] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10204.883597] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10204.898753] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10204.907911] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10204.913303] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10207.712164] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8184s! [10207.715844] Showing busy workqueues and worker pools: [10207.717620] workqueue events: flags=0x0 [10207.718865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10207.726469] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10207.739839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10207.744515] pending: sec_gc_main [ptlrpc] [10207.746416] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10207.751352] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10207.758903] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10207.761976] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10207.768142] workqueue events_long: flags=0x0 [10207.770612] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10207.776195] pending: lu_objects_flush [obdclass] [10207.779333] workqueue events_unbound: flags=0x2 [10207.785613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10207.791970] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10207.799056] workqueue events_power_efficient: flags=0x80 [10207.801040] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10207.803477] pending: neigh_periodic_work, do_cache_clean [10207.805532] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10207.808268] pending: neigh_periodic_work, check_lifetime [10207.810461] workqueue mm_percpu_wq: flags=0x8 [10207.812455] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10207.814815] pending: vmstat_update [10207.828472] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10207.840236] pending: vmstat_update [10207.842752] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10207.853597] pending: vmstat_update [10207.858393] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10207.864688] pending: vmstat_update [10207.865964] workqueue writeback: flags=0x4a [10207.867888] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10207.878585] pending: wb_workfn, wb_workfn [10207.884461] workqueue xprtiod: flags=0xa [10207.886678] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10207.890808] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10207.897724] workqueue ipv6_addrconf: flags=0x40008 [10207.900976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10207.905760] pending: addrconf_verify_work [10207.908015] workqueue knbd0-recv: flags=0x1a [10207.911341] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10207.917550] in-flight: 647:recv_work [10207.919962] workqueue ptlrpc_pinger: flags=0x2 [10207.921631] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10207.927170] pending: ptlrpc_pinger_main [ptlrpc] [10207.928712] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10197s workers=2 idle: 652 [10208.056073] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10208.059334] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10208.061687] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10208.061755] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10208.061762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10208.061765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10208.061778] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10208.061782] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10208.061787] RAX: 00000000000019f9 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10208.061789] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10208.061791] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10208.061793] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10208.061796] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10208.061799] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10208.061802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10208.061804] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10208.061813] Call Trace: [10208.061818] [10208.061823] ? show_regs.cold.9+0x22/0x2f [10208.061833] ? watchdog_timer_fn.cold.6+0x62/0x10a [10208.061838] ? __hrtimer_run_queues+0x1c2/0x4d0 [10208.061842] ? watchdog+0x40/0x40 [10208.061846] ? hrtimer_interrupt+0x128/0x2c0 [10208.061850] ? smp_apic_timer_interrupt+0xa6/0x270 [10208.061858] ? apic_timer_interrupt+0xf/0x20 [10208.061862] [10208.061865] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10208.061871] do_raw_spin_lock+0x177/0x190 [10208.061876] _raw_spin_lock+0x21/0x30 [10208.061881] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10208.062558] ? do_raw_spin_unlock+0x75/0x190 [10208.062564] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10208.062771] ? do_wait_intr+0xf0/0xf0 [10208.062778] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10208.062981] mdc_close+0x53b/0xd70 [mdc] [10208.063019] lmv_close+0x18d/0x490 [lmv] [10208.063051] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10208.063777] ll_md_real_close+0xa5/0x2a0 [lustre] [10208.063856] ll_md_close+0x274/0x800 [lustre] [10208.064012] ll_file_release+0x128/0xc50 [lustre] [10208.064272] __fput+0x10d/0x440 [10208.064282] ____fput+0x12/0x20 [10208.064287] task_work_run+0xc6/0x110 [10208.064294] exit_to_usermode_loop+0x1dd/0x1f0 [10208.064300] do_syscall_64+0x3d6/0x440 [10208.064304] entry_SYSCALL_64_after_hwframe+0x49/0xae [10208.064311] RIP: 0033:0x7f1a63983cd8 [10208.064319] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10208.064322] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10208.064327] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10208.064330] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10208.064332] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10208.064334] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10208.064336] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10208.569067] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10208.625378] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10208.632568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10208.638214] RIP: 0010:native_safe_halt+0x2e/0x50 [10208.640567] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10208.651203] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10208.654716] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10208.658569] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10208.662100] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10208.667319] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10208.679721] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10208.685072] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10208.696226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10208.704215] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10208.714503] Call Trace: [10208.716166] [10208.717602] ? show_regs.cold.9+0x22/0x2f [10208.720446] ? watchdog_timer_fn.cold.6+0x62/0x10a [10208.724158] ? __hrtimer_run_queues+0x1c2/0x4d0 [10208.727447] ? watchdog+0x40/0x40 [10208.729408] ? hrtimer_interrupt+0x128/0x2c0 [10208.731657] ? smp_apic_timer_interrupt+0xa6/0x270 [10208.736555] ? apic_timer_interrupt+0xf/0x20 [10208.739747] [10208.741147] ? native_safe_halt+0x2e/0x50 [10208.744215] kvm_wait+0xcd/0xe0 [10208.745798] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10208.750140] do_raw_spin_lock+0x177/0x190 [10208.755601] _raw_spin_lock+0x21/0x30 [10208.758481] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10208.765555] obd_mod_free+0x46/0x140 [ptlrpc] [10208.769246] mdc_commit_open+0x91/0xb0 [mdc] [10208.774589] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10208.779058] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10208.790140] after_reply+0xe2c/0x1780 [ptlrpc] [10208.796840] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10208.804491] ? __next_timer_interrupt+0x160/0x160 [10208.808547] ptlrpcd+0xa82/0xe40 [ptlrpc] [10208.810355] ? do_wait_intr+0xf0/0xf0 [10208.811715] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10208.829667] kthread+0x1d1/0x200 [10208.835251] ? set_kthread_struct+0x70/0x70 [10208.840625] ret_from_fork+0x1f/0x30 [10228.057071] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10228.064458] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10228.095090] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10228.102189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10228.108203] RIP: 0010:native_safe_halt+0x2e/0x50 [10228.111010] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10228.123088] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10228.127479] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10228.131406] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [10228.136397] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10228.142389] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10228.147144] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10228.152082] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10228.156062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10228.159845] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10228.164132] Call Trace: [10228.165454] [10228.166648] ? show_regs.cold.9+0x22/0x2f [10228.169011] ? watchdog_timer_fn.cold.6+0x62/0x10a [10228.171520] ? __hrtimer_run_queues+0x1c2/0x4d0 [10228.175179] ? watchdog+0x40/0x40 [10228.176954] ? hrtimer_interrupt+0x128/0x2c0 [10228.180067] ? smp_apic_timer_interrupt+0xa6/0x270 [10228.184434] ? apic_timer_interrupt+0xf/0x20 [10228.186980] [10228.188439] ? native_safe_halt+0x2e/0x50 [10228.190409] kvm_wait+0xcd/0xe0 [10228.192161] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10228.195458] do_raw_spin_lock+0x177/0x190 [10228.197243] _raw_spin_lock+0x21/0x30 [10228.199580] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10228.202929] ? do_raw_spin_unlock+0x75/0x190 [10228.205364] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10228.208704] ? do_wait_intr+0xf0/0xf0 [10228.210048] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10228.212588] mdc_close+0x53b/0xd70 [mdc] [10228.214253] lmv_close+0x18d/0x490 [lmv] [10228.216157] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10228.218498] ll_md_real_close+0xa5/0x2a0 [lustre] [10228.220285] ll_md_close+0x274/0x800 [lustre] [10228.221690] ll_file_release+0x128/0xc50 [lustre] [10228.223992] __fput+0x10d/0x440 [10228.225440] ____fput+0x12/0x20 [10228.226885] task_work_run+0xc6/0x110 [10228.228122] exit_to_usermode_loop+0x1dd/0x1f0 [10228.229592] do_syscall_64+0x3d6/0x440 [10228.230936] entry_SYSCALL_64_after_hwframe+0x49/0xae [10228.232556] RIP: 0033:0x7f1a63983cd8 [10228.234187] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10228.240580] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10228.243409] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10228.246317] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10228.249149] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10228.252277] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10228.254661] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10232.053284] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [10232.058327] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10232.084776] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10232.092328] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10232.101160] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10232.118052] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10232.145350] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10232.156191] RAX: 0000000000002986 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10232.161196] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10232.170185] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10232.177069] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10232.191259] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10232.199630] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10232.205496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10232.209413] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10232.213063] Call Trace: [10232.214497] [10232.215952] ? show_regs.cold.9+0x22/0x2f [10232.218171] ? watchdog_timer_fn.cold.6+0x62/0x10a [10232.223045] ? __hrtimer_run_queues+0x1c2/0x4d0 [10232.225993] ? watchdog+0x40/0x40 [10232.228472] ? hrtimer_interrupt+0x128/0x2c0 [10232.235106] ? smp_apic_timer_interrupt+0xa6/0x270 [10232.238228] ? apic_timer_interrupt+0xf/0x20 [10232.242427] [10232.245506] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10232.251763] do_raw_spin_lock+0x177/0x190 [10232.256161] _raw_spin_lock+0x21/0x30 [10232.259214] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10232.264500] ? do_raw_spin_unlock+0x75/0x190 [10232.269157] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10232.274256] ? do_wait_intr+0xf0/0xf0 [10232.275547] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10232.279761] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10232.283287] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10232.286476] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10232.289126] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10232.291800] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10232.294957] mdc_intent_lock+0x353/0x7e0 [mdc] [10232.297200] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10232.300130] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10232.303170] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10232.305786] lmv_intent_open+0x338/0x1150 [lmv] [10232.307744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10232.310077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10232.312478] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10232.314381] ll_intent_lock+0xc5/0xaa0 [lustre] [10232.316177] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10232.317861] ll_lookup_it+0xe0e/0x2270 [lustre] [10232.322112] ? do_raw_spin_unlock+0x75/0x190 [10232.324638] ? _raw_spin_unlock+0x12/0x30 [10232.326851] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10232.330023] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10232.332267] ll_atomic_open+0x36a/0xd70 [lustre] [10232.334892] lookup_open+0x356/0xb40 [10232.336790] do_last+0x313/0xfc0 [10232.338022] ? nd_jump_root+0xe5/0x160 [10232.340011] ? path_init+0x437/0x520 [10232.341878] path_openat+0xf7/0x500 [10232.343933] ? do_raw_spin_unlock+0x75/0x190 [10232.345882] do_filp_open+0x99/0x140 [10232.347515] ? getname_flags+0x6e/0x330 [10232.350995] ? __check_object_size+0xff/0x256 [10232.356121] ? do_raw_spin_unlock+0x75/0x190 [10232.360764] ? _raw_spin_unlock+0x12/0x30 [10232.363189] do_sys_openat2+0x2b4/0x410 [10232.364843] do_sys_open+0x73/0xa0 [10232.366118] __x64_sys_openat+0x24/0x30 [10232.367960] do_syscall_64+0xc1/0x440 [10232.369416] entry_SYSCALL_64_after_hwframe+0x49/0xae [10232.371753] RIP: 0033:0x7f1a63983332 [10232.373214] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10232.382416] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10232.385023] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10232.386995] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10232.389645] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10232.391643] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10232.393518] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10236.056009] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [10236.059267] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [10236.059898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10236.059979] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10236.059985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10236.059988] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10236.060002] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10236.060006] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10236.060012] RAX: 000000000000707a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10236.060014] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10236.060017] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10236.060090] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10236.060092] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10236.060096] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10236.060099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10236.060102] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10236.060111] Call Trace: [10236.060116] [10236.060121] ? show_regs.cold.9+0x22/0x2f [10236.060131] ? watchdog_timer_fn.cold.6+0x62/0x10a [10236.060138] ? __hrtimer_run_queues+0x1c2/0x4d0 [10236.060142] ? watchdog+0x40/0x40 [10236.060147] ? hrtimer_interrupt+0x128/0x2c0 [10236.060151] ? smp_apic_timer_interrupt+0xa6/0x270 [10236.060159] ? apic_timer_interrupt+0xf/0x20 [10236.060169] [10236.060172] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10236.060179] do_raw_spin_lock+0x177/0x190 [10236.060184] _raw_spin_lock+0x21/0x30 [10236.060188] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10236.060788] ? do_raw_spin_unlock+0x75/0x190 [10236.060794] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10236.061041] ? do_wait_intr+0xf0/0xf0 [10236.061048] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10236.061268] mdc_close+0x53b/0xd70 [mdc] [10236.061311] lmv_close+0x18d/0x490 [lmv] [10236.061338] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10236.061704] ll_md_real_close+0xa5/0x2a0 [lustre] [10236.061808] ll_md_close+0x274/0x800 [lustre] [10236.061910] ll_file_release+0x128/0xc50 [lustre] [10236.062006] __fput+0x10d/0x440 [10236.062015] ____fput+0x12/0x20 [10236.062029] task_work_run+0xc6/0x110 [10236.062036] exit_to_usermode_loop+0x1dd/0x1f0 [10236.062043] do_syscall_64+0x3d6/0x440 [10236.062047] entry_SYSCALL_64_after_hwframe+0x49/0xae [10236.062055] RIP: 0033:0x7f1a63983cd8 [10236.062062] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10236.062066] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10236.062071] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10236.062074] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10236.062076] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10236.062078] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10236.062081] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10236.434199] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10236.480897] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10236.489830] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10236.495923] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10236.506785] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10236.534159] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10236.539851] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10236.564213] RDX: 0000000000003063 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10236.580648] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10236.586093] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10236.596700] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10236.600524] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10236.612239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10236.620438] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10236.632578] Call Trace: [10236.634218] [10236.636272] ? show_regs.cold.9+0x22/0x2f [10236.639313] ? watchdog_timer_fn.cold.6+0x62/0x10a [10236.642425] ? __hrtimer_run_queues+0x1c2/0x4d0 [10236.647271] ? watchdog+0x40/0x40 [10236.652624] ? hrtimer_interrupt+0x128/0x2c0 [10236.657165] ? smp_apic_timer_interrupt+0xa6/0x270 [10236.660068] ? apic_timer_interrupt+0xf/0x20 [10236.668526] [10236.670861] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10236.674835] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10236.683372] do_raw_spin_lock+0x177/0x190 [10236.689090] _raw_spin_lock+0x21/0x30 [10236.696347] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10236.704774] obd_mod_free+0x46/0x140 [ptlrpc] [10236.706705] mdc_commit_open+0x91/0xb0 [mdc] [10236.715331] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10236.725898] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10236.732152] after_reply+0xe2c/0x1780 [ptlrpc] [10236.736273] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10236.745410] ? __next_timer_interrupt+0x160/0x160 [10236.748433] ptlrpcd+0xa82/0xe40 [ptlrpc] [10236.750832] ? do_wait_intr+0xf0/0xf0 [10236.754688] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10236.761619] kthread+0x1d1/0x200 [10236.765798] ? set_kthread_struct+0x70/0x70 [10236.773904] ret_from_fork+0x1f/0x30 [10238.435075] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8215s! [10238.439079] Showing busy workqueues and worker pools: [10238.442059] workqueue events: flags=0x0 [10238.444243] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10238.447338] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10238.451745] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10238.455400] pending: sec_gc_main [ptlrpc] [10238.457642] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10238.460993] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10238.466064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10238.469744] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10238.474728] workqueue events_long: flags=0x0 [10238.476377] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10238.480323] pending: lu_objects_flush [obdclass] [10238.482864] workqueue events_unbound: flags=0x2 [10238.485731] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10238.488308] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10238.493572] workqueue events_power_efficient: flags=0x80 [10238.496050] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10238.499460] pending: neigh_periodic_work, do_cache_clean [10238.502423] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10238.505788] pending: neigh_periodic_work, check_lifetime [10238.508568] workqueue mm_percpu_wq: flags=0x8 [10238.510457] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10238.513819] pending: vmstat_update [10238.515777] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10238.518678] pending: vmstat_update [10238.520586] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10238.523464] pending: vmstat_update [10238.525403] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10238.528726] pending: vmstat_update [10238.530842] workqueue writeback: flags=0x4a [10238.533159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10238.535809] pending: wb_workfn, wb_workfn [10238.538183] workqueue xprtiod: flags=0xa [10238.539801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10238.542267] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10238.544172] workqueue ipv6_addrconf: flags=0x40008 [10238.546015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10238.548126] pending: addrconf_verify_work [10238.549356] workqueue knbd0-recv: flags=0x1a [10238.550982] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10238.553838] in-flight: 647:recv_work [10238.555185] workqueue ptlrpc_pinger: flags=0x2 [10238.556504] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10238.559289] pending: ptlrpc_pinger_main [ptlrpc] [10238.561291] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10228s workers=2 idle: 652 [10256.058769] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10256.067937] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10256.127232] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10256.145795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10256.157489] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10256.163938] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10256.186170] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10256.200754] RAX: 0000000000007f8e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10256.211232] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10256.224106] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10256.228412] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10256.236524] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10256.249214] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10256.260894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10256.272882] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10256.282622] Call Trace: [10256.288790] [10256.290006] ? show_regs.cold.9+0x22/0x2f [10256.292758] ? watchdog_timer_fn.cold.6+0x62/0x10a [10256.302685] ? __hrtimer_run_queues+0x1c2/0x4d0 [10256.312301] ? watchdog+0x40/0x40 [10256.318550] ? hrtimer_interrupt+0x128/0x2c0 [10256.324945] ? smp_apic_timer_interrupt+0xa6/0x270 [10256.330973] ? apic_timer_interrupt+0xf/0x20 [10256.337698] [10256.339751] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10256.351406] do_raw_spin_lock+0x177/0x190 [10256.354820] _raw_spin_lock+0x21/0x30 [10256.357856] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10256.367304] ? do_raw_spin_unlock+0x75/0x190 [10256.377376] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10256.382646] ? do_wait_intr+0xf0/0xf0 [10256.391378] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10256.398078] mdc_close+0x53b/0xd70 [mdc] [10256.402831] lmv_close+0x18d/0x490 [lmv] [10256.406638] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10256.410964] ll_md_real_close+0xa5/0x2a0 [lustre] [10256.414573] ll_md_close+0x274/0x800 [lustre] [10256.417959] ll_file_release+0x128/0xc50 [lustre] [10256.427377] __fput+0x10d/0x440 [10256.430135] ____fput+0x12/0x20 [10256.432615] task_work_run+0xc6/0x110 [10256.440314] exit_to_usermode_loop+0x1dd/0x1f0 [10256.442564] do_syscall_64+0x3d6/0x440 [10256.443845] entry_SYSCALL_64_after_hwframe+0x49/0xae [10256.453898] RIP: 0033:0x7f1a63983cd8 [10256.457330] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10256.479003] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10256.488805] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10256.501650] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10256.511503] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10256.524341] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10256.535953] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10264.054465] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [10264.056058] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [10264.058245] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10264.058325] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10264.058332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10264.058335] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10264.058351] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10264.059184] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [10264.058355] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10264.061719] RAX: 0000000000006f10 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10264.061722] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10264.061725] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10264.061727] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10264.061729] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10264.061733] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10264.061737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10264.061739] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10264.061748] Call Trace: [10264.061754] [10264.061760] ? show_regs.cold.9+0x22/0x2f [10264.061774] ? watchdog_timer_fn.cold.6+0x62/0x10a [10264.061783] ? __hrtimer_run_queues+0x1c2/0x4d0 [10264.061788] ? watchdog+0x40/0x40 [10264.061792] ? hrtimer_interrupt+0x128/0x2c0 [10264.061990] ? smp_apic_timer_interrupt+0xa6/0x270 [10264.061998] ? apic_timer_interrupt+0xf/0x20 [10264.062004] [10264.062007] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10264.062017] do_raw_spin_lock+0x177/0x190 [10264.062022] _raw_spin_lock+0x21/0x30 [10264.062028] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10264.063561] ? do_raw_spin_unlock+0x75/0x190 [10264.063569] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10264.063766] ? do_wait_intr+0xf0/0xf0 [10264.063774] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10264.065674] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10264.065935] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10264.067178] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10264.067433] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10264.067444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10264.067448] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10264.067461] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10264.067466] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10264.067471] RAX: 0000000000000a09 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10264.067343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10264.067474] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10264.067534] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10264.067540] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10264.067543] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10264.067547] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10264.067550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10264.067552] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10264.067560] Call Trace: [10264.067566] [10264.067539] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10264.067569] ? show_regs.cold.9+0x22/0x2f [10264.067581] ? watchdog_timer_fn.cold.6+0x62/0x10a [10264.067591] ? __hrtimer_run_queues+0x1c2/0x4d0 [10264.067595] ? watchdog+0x40/0x40 [10264.067587] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10264.067599] ? hrtimer_interrupt+0x128/0x2c0 [10264.067606] ? smp_apic_timer_interrupt+0xa6/0x270 [10264.067614] ? apic_timer_interrupt+0xf/0x20 [10264.067623] [10264.067627] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10264.067634] do_raw_spin_lock+0x177/0x190 [10264.067624] mdc_intent_lock+0x353/0x7e0 [mdc] [10264.067640] _raw_spin_lock+0x21/0x30 [10264.067681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10264.067647] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10264.068577] ? do_raw_spin_unlock+0x75/0x190 [10264.068585] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10264.068928] ? do_wait_intr+0xf0/0xf0 [10264.067791] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10264.069008] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10264.069468] mdc_close+0x53b/0xd70 [mdc] [10264.069590] lmv_close+0x18d/0x490 [lmv] [10264.069625] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10264.069974] ll_md_real_close+0xa5/0x2a0 [lustre] [10264.070058] ll_md_close+0x274/0x800 [lustre] [10264.070142] ll_file_release+0x128/0xc50 [lustre] [10264.070576] __fput+0x10d/0x440 [10264.070587] ____fput+0x12/0x20 [10264.070592] task_work_run+0xc6/0x110 [10264.070600] exit_to_usermode_loop+0x1dd/0x1f0 [10264.070607] do_syscall_64+0x3d6/0x440 [10264.070612] entry_SYSCALL_64_after_hwframe+0x49/0xae [10264.070620] RIP: 0033:0x7f1a63983cd8 [10264.070629] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10264.070633] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10264.070638] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10264.070641] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10264.070643] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10264.070645] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10264.070648] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10264.106561] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10264.106638] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10264.106646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10264.106649] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10264.106665] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10264.106670] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10264.106674] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10264.106677] RDX: 00000000000051eb RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10264.106679] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10264.106682] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10264.106684] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10264.106687] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10264.106690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10264.106850] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10264.106861] Call Trace: [10264.106866] [10264.106872] ? show_regs.cold.9+0x22/0x2f [10264.106881] ? watchdog_timer_fn.cold.6+0x62/0x10a [10264.106888] ? __hrtimer_run_queues+0x1c2/0x4d0 [10264.106892] ? watchdog+0x40/0x40 [10264.106897] ? hrtimer_interrupt+0x128/0x2c0 [10264.106901] ? smp_apic_timer_interrupt+0xa6/0x270 [10264.106909] ? apic_timer_interrupt+0xf/0x20 [10264.106913] [10264.106921] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10264.106927] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10264.106932] do_raw_spin_lock+0x177/0x190 [10264.106936] _raw_spin_lock+0x21/0x30 [10264.106941] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10264.107225] obd_mod_free+0x46/0x140 [ptlrpc] [10264.107960] mdc_commit_open+0x91/0xb0 [mdc] [10264.107996] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10264.108363] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10264.108588] after_reply+0xe2c/0x1780 [ptlrpc] [10264.109073] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10264.109396] ? __next_timer_interrupt+0x160/0x160 [10264.109406] ptlrpcd+0xa82/0xe40 [ptlrpc] [10264.109857] ? do_wait_intr+0xf0/0xf0 [10264.109868] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10264.110097] kthread+0x1d1/0x200 [10264.110103] ? set_kthread_struct+0x70/0x70 [10264.110108] ret_from_fork+0x1f/0x30 [10264.968417] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10264.973655] lmv_intent_open+0x338/0x1150 [lmv] [10264.976275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10264.982344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10264.989500] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10264.991449] ll_intent_lock+0xc5/0xaa0 [lustre] [10264.997518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10265.002511] ll_lookup_it+0xe0e/0x2270 [lustre] [10265.007626] ? do_raw_spin_unlock+0x75/0x190 [10265.010719] ? _raw_spin_unlock+0x12/0x30 [10265.013324] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10265.020713] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10265.025681] ll_atomic_open+0x36a/0xd70 [lustre] [10265.028780] lookup_open+0x356/0xb40 [10265.030427] do_last+0x313/0xfc0 [10265.037523] ? nd_jump_root+0xe5/0x160 [10265.046808] ? path_init+0x437/0x520 [10265.050196] path_openat+0xf7/0x500 [10265.056636] ? do_raw_spin_unlock+0x75/0x190 [10265.060999] do_filp_open+0x99/0x140 [10265.065280] ? getname_flags+0x6e/0x330 [10265.067164] ? __check_object_size+0xff/0x256 [10265.071737] ? do_raw_spin_unlock+0x75/0x190 [10265.075873] ? _raw_spin_unlock+0x12/0x30 [10265.077573] do_sys_openat2+0x2b4/0x410 [10265.082218] do_sys_open+0x73/0xa0 [10265.086108] __x64_sys_openat+0x24/0x30 [10265.087809] do_syscall_64+0xc1/0x440 [10265.090790] entry_SYSCALL_64_after_hwframe+0x49/0xae [10265.095791] RIP: 0033:0x7f1a63983332 [10265.098722] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10265.116059] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10265.125001] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10265.133314] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10265.138606] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10265.144792] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10265.152932] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10268.643423] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8245s! [10268.650339] Showing busy workqueues and worker pools: [10268.656430] workqueue events: flags=0x0 [10268.666964] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10268.683437] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10268.689981] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10268.705492] pending: sec_gc_main [ptlrpc] [10268.710856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10268.719717] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10268.734643] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10268.736321] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10268.743923] workqueue events_long: flags=0x0 [10268.748505] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10268.752007] pending: lu_objects_flush [obdclass] [10268.757310] workqueue events_unbound: flags=0x2 [10268.766297] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10268.774475] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10268.787804] workqueue events_power_efficient: flags=0x80 [10268.794962] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10268.799818] pending: neigh_periodic_work, do_cache_clean [10268.810227] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10268.817923] pending: neigh_periodic_work, check_lifetime [10268.826723] workqueue mm_percpu_wq: flags=0x8 [10268.838867] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10268.848191] pending: vmstat_update [10268.849954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10268.856382] pending: vmstat_update [10268.864063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10268.866627] pending: vmstat_update [10268.871783] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10268.877299] pending: vmstat_update [10268.882817] workqueue writeback: flags=0x4a [10268.887789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10268.893749] pending: wb_workfn, wb_workfn [10268.896238] workqueue xprtiod: flags=0xa [10268.898693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10268.905250] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10268.913276] workqueue ipv6_addrconf: flags=0x40008 [10268.916754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10268.934846] pending: addrconf_verify_work [10268.937299] workqueue knbd0-recv: flags=0x1a [10268.943939] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10268.958822] in-flight: 647:recv_work [10268.961715] workqueue ptlrpc_pinger: flags=0x2 [10268.965505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10268.969794] pending: ptlrpc_pinger_main [ptlrpc] [10268.977611] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10259s workers=2 idle: 652 [10284.057009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10284.068464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10284.099630] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10284.104859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10284.111791] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10284.116800] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10284.130757] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10284.136379] RAX: 0000000000005a55 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10284.140009] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10284.143047] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10284.150028] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10284.155450] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10284.163448] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10284.167574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10284.172138] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10284.175945] Call Trace: [10284.177286] [10284.178395] ? show_regs.cold.9+0x22/0x2f [10284.179959] ? watchdog_timer_fn.cold.6+0x62/0x10a [10284.182154] ? __hrtimer_run_queues+0x1c2/0x4d0 [10284.184420] ? watchdog+0x40/0x40 [10284.185894] ? hrtimer_interrupt+0x128/0x2c0 [10284.187548] ? smp_apic_timer_interrupt+0xa6/0x270 [10284.193271] ? apic_timer_interrupt+0xf/0x20 [10284.195102] [10284.198429] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10284.201918] do_raw_spin_lock+0x177/0x190 [10284.205082] _raw_spin_lock+0x21/0x30 [10284.207356] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10284.211087] ? do_raw_spin_unlock+0x75/0x190 [10284.213139] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10284.220707] ? do_wait_intr+0xf0/0xf0 [10284.223522] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10284.228024] mdc_close+0x53b/0xd70 [mdc] [10284.229803] lmv_close+0x18d/0x490 [lmv] [10284.231053] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10284.234061] ll_md_real_close+0xa5/0x2a0 [lustre] [10284.236795] ll_md_close+0x274/0x800 [lustre] [10284.242989] ll_file_release+0x128/0xc50 [lustre] [10284.245753] __fput+0x10d/0x440 [10284.247403] ____fput+0x12/0x20 [10284.249396] task_work_run+0xc6/0x110 [10284.251258] exit_to_usermode_loop+0x1dd/0x1f0 [10284.253367] do_syscall_64+0x3d6/0x440 [10284.255018] entry_SYSCALL_64_after_hwframe+0x49/0xae [10284.257131] RIP: 0033:0x7f1a63983cd8 [10284.262641] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10284.271752] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10284.275149] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10284.278355] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10284.281431] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10284.288747] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10284.295883] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10292.057163] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [10292.058462] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [10292.061514] Modules linked in: lustre(O) [10292.065836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [10292.067874] osp(O) ofd(O) lod(O) mdt(O) [10292.077596] lfsck(O) obdecho(O) mgc(O) [10292.079626] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10292.079695] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10292.079701] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10292.079703] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10292.079715] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10292.079718] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10292.079721] RAX: 0000000000001a17 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10292.079723] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10292.079724] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10292.079726] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10292.079727] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10292.079730] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10292.079732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10292.079734] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10292.079740] Call Trace: [10292.079745] [10292.079749] ? show_regs.cold.9+0x22/0x2f [10292.079757] ? watchdog_timer_fn.cold.6+0x62/0x10a [10292.079761] ? __hrtimer_run_queues+0x1c2/0x4d0 [10292.079764] ? watchdog+0x40/0x40 [10292.079767] ? hrtimer_interrupt+0x128/0x2c0 [10292.079769] ? smp_apic_timer_interrupt+0xa6/0x270 [10292.079776] ? apic_timer_interrupt+0xf/0x20 [10292.079779] [10292.079781] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10292.079785] do_raw_spin_lock+0x177/0x190 [10292.079788] _raw_spin_lock+0x21/0x30 [10292.079792] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10292.080223] ? do_raw_spin_unlock+0x75/0x190 [10292.080229] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10292.082420] mdc(O) [10292.103592] ? do_wait_intr+0xf0/0xf0 [10292.111871] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [10292.120480] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10292.120768] mdc_close+0x53b/0xd70 [mdc] [10292.121215] lmv_close+0x18d/0x490 [lmv] [10292.121245] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10292.121457] ll_md_real_close+0xa5/0x2a0 [lustre] [10292.121539] ll_md_close+0x274/0x800 [lustre] [10292.121629] ll_file_release+0x128/0xc50 [lustre] [10292.121717] __fput+0x10d/0x440 [10292.121728] ____fput+0x12/0x20 [10292.121733] task_work_run+0xc6/0x110 [10292.121741] exit_to_usermode_loop+0x1dd/0x1f0 [10292.121749] do_syscall_64+0x3d6/0x440 [10292.121754] entry_SYSCALL_64_after_hwframe+0x49/0xae [10292.121764] RIP: 0033:0x7f1a63983cd8 [10292.121772] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10292.121776] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10292.121781] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10292.121783] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10292.121786] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10292.121788] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10292.121790] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10292.291984] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10292.314182] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10292.321587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10292.326964] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10292.331525] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10292.342850] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10292.347559] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10292.353423] RDX: 00000000000067fc RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10292.358378] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10292.361969] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10292.365510] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10292.368745] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10292.373673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10292.377721] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10292.381290] Call Trace: [10292.382605] [10292.383483] ? show_regs.cold.9+0x22/0x2f [10292.386771] ? watchdog_timer_fn.cold.6+0x62/0x10a [10292.388939] ? __hrtimer_run_queues+0x1c2/0x4d0 [10292.391353] ? watchdog+0x40/0x40 [10292.392697] ? hrtimer_interrupt+0x128/0x2c0 [10292.395494] ? smp_apic_timer_interrupt+0xa6/0x270 [10292.397608] ? apic_timer_interrupt+0xf/0x20 [10292.399367] [10292.400341] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10292.402992] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10292.405844] do_raw_spin_lock+0x177/0x190 [10292.407659] _raw_spin_lock+0x21/0x30 [10292.410521] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10292.413812] obd_mod_free+0x46/0x140 [ptlrpc] [10292.416470] mdc_commit_open+0x91/0xb0 [mdc] [10292.418690] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10292.421249] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10292.424452] after_reply+0xe2c/0x1780 [ptlrpc] [10292.426707] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10292.429584] ? __next_timer_interrupt+0x160/0x160 [10292.432151] ptlrpcd+0xa82/0xe40 [ptlrpc] [10292.434689] ? do_wait_intr+0xf0/0xf0 [10292.436764] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10292.439382] kthread+0x1d1/0x200 [10292.441434] ? set_kthread_struct+0x70/0x70 [10292.443535] ret_from_fork+0x1f/0x30 [10296.053013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10296.062190] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10296.094468] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10296.102032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10296.105000] RIP: 0010:native_safe_halt+0x2e/0x50 [10296.110060] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10296.122561] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10296.131811] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10296.136991] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10296.141004] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10296.145542] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10296.149989] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10296.156748] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10296.167201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10296.175563] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10296.182594] Call Trace: [10296.183617] [10296.187181] ? show_regs.cold.9+0x22/0x2f [10296.189872] ? watchdog_timer_fn.cold.6+0x62/0x10a [10296.194026] ? __hrtimer_run_queues+0x1c2/0x4d0 [10296.196861] ? watchdog+0x40/0x40 [10296.198618] ? hrtimer_interrupt+0x128/0x2c0 [10296.201279] ? smp_apic_timer_interrupt+0xa6/0x270 [10296.203543] ? apic_timer_interrupt+0xf/0x20 [10296.205566] [10296.206570] ? native_safe_halt+0x2e/0x50 [10296.208167] kvm_wait+0xcd/0xe0 [10296.209203] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10296.214629] do_raw_spin_lock+0x177/0x190 [10296.216881] _raw_spin_lock+0x21/0x30 [10296.219730] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10296.231449] ? do_raw_spin_unlock+0x75/0x190 [10296.236408] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10296.241836] ? do_wait_intr+0xf0/0xf0 [10296.243177] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10296.255265] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10296.261480] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10296.266872] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10296.271530] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10296.275886] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10296.279094] mdc_intent_lock+0x353/0x7e0 [mdc] [10296.284820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10296.287382] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10296.291052] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10296.297226] lmv_intent_open+0x338/0x1150 [lmv] [10296.303000] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10296.311926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10296.318045] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10296.321046] ll_intent_lock+0xc5/0xaa0 [lustre] [10296.322970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10296.327895] ll_lookup_it+0xe0e/0x2270 [lustre] [10296.337896] ? do_raw_spin_unlock+0x75/0x190 [10296.344425] ? _raw_spin_unlock+0x12/0x30 [10296.349219] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10296.359195] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10296.364978] ll_atomic_open+0x36a/0xd70 [lustre] [10296.368269] lookup_open+0x356/0xb40 [10296.376737] do_last+0x313/0xfc0 [10296.383730] ? nd_jump_root+0xe5/0x160 [10296.386086] ? path_init+0x437/0x520 [10296.388046] path_openat+0xf7/0x500 [10296.395559] ? do_raw_spin_unlock+0x75/0x190 [10296.404147] do_filp_open+0x99/0x140 [10296.407486] ? getname_flags+0x6e/0x330 [10296.410434] ? __check_object_size+0xff/0x256 [10296.417644] ? do_raw_spin_unlock+0x75/0x190 [10296.420818] ? _raw_spin_unlock+0x12/0x30 [10296.422628] do_sys_openat2+0x2b4/0x410 [10296.428460] do_sys_open+0x73/0xa0 [10296.429742] __x64_sys_openat+0x24/0x30 [10296.436316] do_syscall_64+0xc1/0x440 [10296.442969] entry_SYSCALL_64_after_hwframe+0x49/0xae [10296.447815] RIP: 0033:0x7f1a63983332 [10296.451342] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10296.468824] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10296.477231] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10296.485847] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10296.489640] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10296.492909] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10296.497415] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10299.361244] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8276s! [10299.365765] Showing busy workqueues and worker pools: [10299.368415] workqueue events: flags=0x0 [10299.369873] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10299.372303] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10299.375212] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10299.380484] pending: sec_gc_main [ptlrpc] [10299.384635] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10299.389642] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10299.394331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10299.397765] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10299.401855] workqueue events_long: flags=0x0 [10299.403607] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10299.406276] pending: lu_objects_flush [obdclass] [10299.408658] workqueue events_unbound: flags=0x2 [10299.410468] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10299.413148] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10299.417501] workqueue events_power_efficient: flags=0x80 [10299.419600] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10299.422386] pending: neigh_periodic_work, do_cache_clean [10299.424659] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10299.426694] pending: neigh_periodic_work, check_lifetime [10299.428588] workqueue mm_percpu_wq: flags=0x8 [10299.429898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10299.432169] pending: vmstat_update [10299.433440] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10299.435594] pending: vmstat_update [10299.436752] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10299.438941] pending: vmstat_update [10299.440211] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10299.442233] pending: vmstat_update [10299.443363] workqueue writeback: flags=0x4a [10299.444691] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10299.446766] pending: wb_workfn, wb_workfn [10299.448286] workqueue xprtiod: flags=0xa [10299.449641] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10299.451659] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10299.454402] workqueue ipv6_addrconf: flags=0x40008 [10299.455938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10299.458264] pending: addrconf_verify_work [10299.459744] workqueue knbd0-recv: flags=0x1a [10299.460883] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10299.463219] in-flight: 647:recv_work [10299.464779] workqueue ptlrpc_pinger: flags=0x2 [10299.466511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10299.468688] pending: ptlrpc_pinger_main [ptlrpc] [10299.471331] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10289s workers=2 idle: 652 [10312.059459] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10312.067803] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10312.136722] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10312.145703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10312.151847] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10312.158485] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10312.173975] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10312.177183] RAX: 0000000000006f8e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10312.180191] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10312.183630] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10312.191059] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10312.195076] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10312.199081] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10312.204073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10312.205937] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10312.213062] Call Trace: [10312.213840] [10312.214502] ? show_regs.cold.9+0x22/0x2f [10312.224102] ? watchdog_timer_fn.cold.6+0x62/0x10a [10312.228851] ? __hrtimer_run_queues+0x1c2/0x4d0 [10312.231343] ? watchdog+0x40/0x40 [10312.232525] ? hrtimer_interrupt+0x128/0x2c0 [10312.234270] ? smp_apic_timer_interrupt+0xa6/0x270 [10312.236093] ? apic_timer_interrupt+0xf/0x20 [10312.238749] [10312.241110] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10312.243861] do_raw_spin_lock+0x177/0x190 [10312.245235] _raw_spin_lock+0x21/0x30 [10312.246566] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10312.248837] ? do_raw_spin_unlock+0x75/0x190 [10312.250438] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10312.252346] ? do_wait_intr+0xf0/0xf0 [10312.253604] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10312.255625] mdc_close+0x53b/0xd70 [mdc] [10312.257113] lmv_close+0x18d/0x490 [lmv] [10312.258466] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10312.260470] ll_md_real_close+0xa5/0x2a0 [lustre] [10312.262468] ll_md_close+0x274/0x800 [lustre] [10312.264106] ll_file_release+0x128/0xc50 [lustre] [10312.265855] __fput+0x10d/0x440 [10312.267034] ____fput+0x12/0x20 [10312.268170] task_work_run+0xc6/0x110 [10312.269604] exit_to_usermode_loop+0x1dd/0x1f0 [10312.271282] do_syscall_64+0x3d6/0x440 [10312.272601] entry_SYSCALL_64_after_hwframe+0x49/0xae [10312.274812] RIP: 0033:0x7f1a63983cd8 [10312.276012] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10312.286837] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10312.294072] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10312.300235] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10312.305377] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10312.308087] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10312.320378] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10320.058685] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [10320.059169] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [10320.066420] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10320.066496] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10320.066545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10320.066548] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10320.066563] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10320.066567] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10320.066572] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10320.066575] RDX: 00000000000062fd RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10320.066577] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10320.066580] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10320.066582] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10320.066585] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10320.066589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10320.066591] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10320.066600] Call Trace: [10320.066606] [10320.066612] ? show_regs.cold.9+0x22/0x2f [10320.066621] ? watchdog_timer_fn.cold.6+0x62/0x10a [10320.066634] ? __hrtimer_run_queues+0x1c2/0x4d0 [10320.066639] ? watchdog+0x40/0x40 [10320.066644] ? hrtimer_interrupt+0x128/0x2c0 [10320.066648] ? smp_apic_timer_interrupt+0xa6/0x270 [10320.066658] ? apic_timer_interrupt+0xf/0x20 [10320.066662] [10320.066665] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10320.066671] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10320.066675] do_raw_spin_lock+0x177/0x190 [10320.066681] _raw_spin_lock+0x21/0x30 [10320.066686] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10320.066991] obd_mod_free+0x46/0x140 [ptlrpc] [10320.073095] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10320.074981] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10320.074988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10320.074991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10320.075006] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10320.075010] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10320.075015] RAX: 0000000000007e86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10320.075017] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10320.075019] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10320.075021] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10320.075024] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10320.075027] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10320.075030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10320.075033] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10320.075041] Call Trace: [10320.075336] [10320.075343] ? show_regs.cold.9+0x22/0x2f [10320.075353] ? watchdog_timer_fn.cold.6+0x62/0x10a [10320.075360] ? __hrtimer_run_queues+0x1c2/0x4d0 [10320.075365] ? watchdog+0x40/0x40 [10320.075369] ? hrtimer_interrupt+0x128/0x2c0 [10320.075374] ? smp_apic_timer_interrupt+0xa6/0x270 [10320.075383] ? apic_timer_interrupt+0xf/0x20 [10320.075388] [10320.075391] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10320.075397] do_raw_spin_lock+0x177/0x190 [10320.075402] _raw_spin_lock+0x21/0x30 [10320.075407] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10320.075686] ? do_raw_spin_unlock+0x75/0x190 [10320.075693] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10320.076990] ? do_wait_intr+0xf0/0xf0 [10320.077001] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10320.077376] mdc_close+0x53b/0xd70 [mdc] [10320.077414] lmv_close+0x18d/0x490 [lmv] [10320.077453] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10320.077567] ll_md_real_close+0xa5/0x2a0 [lustre] [10320.077651] ll_md_close+0x274/0x800 [lustre] [10320.077735] ll_file_release+0x128/0xc50 [lustre] [10320.077813] __fput+0x10d/0x440 [10320.077865] ____fput+0x12/0x20 [10320.077870] task_work_run+0xc6/0x110 [10320.077878] exit_to_usermode_loop+0x1dd/0x1f0 [10320.077885] do_syscall_64+0x3d6/0x440 [10320.077890] entry_SYSCALL_64_after_hwframe+0x49/0xae [10320.077898] RIP: 0033:0x7f1a63983cd8 [10320.077907] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10320.077911] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10320.077916] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10320.077919] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10320.077921] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10320.077924] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10320.077926] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10320.613251] mdc_commit_open+0x91/0xb0 [mdc] [10320.614949] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10320.618970] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10320.624185] after_reply+0xe2c/0x1780 [ptlrpc] [10320.627741] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10320.633176] ? __next_timer_interrupt+0x160/0x160 [10320.637777] ptlrpcd+0xa82/0xe40 [ptlrpc] [10320.642710] ? do_wait_intr+0xf0/0xf0 [10320.645496] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10320.652559] kthread+0x1d1/0x200 [10320.655545] ? set_kthread_struct+0x70/0x70 [10320.659358] ret_from_fork+0x1f/0x30 [10324.053027] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [10324.057689] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10324.078509] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10324.082977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10324.085655] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10324.087567] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10324.093514] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10324.095933] RAX: 00000000000067e3 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10324.098199] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10324.100361] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10324.102466] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10324.104663] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10324.107135] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10324.109528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10324.111384] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10324.113638] Call Trace: [10324.114629] [10324.115476] ? show_regs.cold.9+0x22/0x2f [10324.116911] ? watchdog_timer_fn.cold.6+0x62/0x10a [10324.118780] ? __hrtimer_run_queues+0x1c2/0x4d0 [10324.120363] ? watchdog+0x40/0x40 [10324.121563] ? hrtimer_interrupt+0x128/0x2c0 [10324.123097] ? smp_apic_timer_interrupt+0xa6/0x270 [10324.124743] ? apic_timer_interrupt+0xf/0x20 [10324.126222] [10324.127210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10324.129502] do_raw_spin_lock+0x177/0x190 [10324.130907] _raw_spin_lock+0x21/0x30 [10324.132039] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10324.134035] ? do_raw_spin_unlock+0x75/0x190 [10324.135874] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10324.139619] ? do_wait_intr+0xf0/0xf0 [10324.141061] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10324.143226] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10324.145279] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10324.147694] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10324.152700] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10324.155285] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10324.157866] mdc_intent_lock+0x353/0x7e0 [mdc] [10324.159514] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10324.174058] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10324.183007] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10324.187712] lmv_intent_open+0x338/0x1150 [lmv] [10324.192554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10324.195205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10324.197861] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10324.199632] ll_intent_lock+0xc5/0xaa0 [lustre] [10324.202428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10324.205429] ll_lookup_it+0xe0e/0x2270 [lustre] [10324.207958] ? do_raw_spin_unlock+0x75/0x190 [10324.209882] ? _raw_spin_unlock+0x12/0x30 [10324.212012] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10324.213910] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10324.215845] ll_atomic_open+0x36a/0xd70 [lustre] [10324.218383] lookup_open+0x356/0xb40 [10324.220261] do_last+0x313/0xfc0 [10324.222089] ? nd_jump_root+0xe5/0x160 [10324.223497] ? path_init+0x437/0x520 [10324.224932] path_openat+0xf7/0x500 [10324.226304] ? do_raw_spin_unlock+0x75/0x190 [10324.228697] do_filp_open+0x99/0x140 [10324.230616] ? getname_flags+0x6e/0x330 [10324.232662] ? __check_object_size+0xff/0x256 [10324.234698] ? do_raw_spin_unlock+0x75/0x190 [10324.237022] ? _raw_spin_unlock+0x12/0x30 [10324.238973] do_sys_openat2+0x2b4/0x410 [10324.240824] do_sys_open+0x73/0xa0 [10324.242343] __x64_sys_openat+0x24/0x30 [10324.244319] do_syscall_64+0xc1/0x440 [10324.246141] entry_SYSCALL_64_after_hwframe+0x49/0xae [10324.248349] RIP: 0033:0x7f1a63983332 [10324.250354] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10324.259149] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10324.262526] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10324.265718] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10324.268867] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10324.272168] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10324.275261] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10329.570043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8306s! [10329.574812] Showing busy workqueues and worker pools: [10329.577333] workqueue events: flags=0x0 [10329.579570] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10329.582896] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10329.585726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10329.587899] pending: sec_gc_main [ptlrpc] [10329.590807] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10329.593427] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10329.596336] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10329.598460] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10329.602932] workqueue events_long: flags=0x0 [10329.605098] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10329.607872] pending: lu_objects_flush [obdclass] [10329.609923] workqueue events_unbound: flags=0x2 [10329.611320] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10329.613674] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10329.617467] workqueue events_power_efficient: flags=0x80 [10329.620011] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10329.623346] pending: neigh_periodic_work, do_cache_clean [10329.625419] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10329.628338] pending: neigh_periodic_work, check_lifetime [10329.630972] workqueue mm_percpu_wq: flags=0x8 [10329.632949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10329.636109] pending: vmstat_update [10329.637672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10329.640809] pending: vmstat_update [10329.642421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10329.645316] pending: vmstat_update [10329.647223] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10329.650628] pending: vmstat_update [10329.652493] workqueue writeback: flags=0x4a [10329.654759] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10329.658201] pending: wb_workfn, wb_workfn [10329.660406] workqueue xprtiod: flags=0xa [10329.661909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10329.664210] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10329.667717] workqueue ipv6_addrconf: flags=0x40008 [10329.669489] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10329.672974] pending: addrconf_verify_work [10329.674792] workqueue knbd0-recv: flags=0x1a [10329.676278] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10329.679599] in-flight: 647:recv_work [10329.681498] workqueue ptlrpc_pinger: flags=0x2 [10329.683986] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10329.687169] pending: ptlrpc_pinger_main [ptlrpc] [10329.690027] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10319s workers=2 idle: 652 [10340.060297] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10340.067585] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10340.118068] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10340.127748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10340.133052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10340.136268] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10340.159076] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10340.163169] RAX: 0000000000006689 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10340.170853] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10340.179726] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10340.190243] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10340.192435] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10340.202486] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10340.209188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10340.211730] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10340.216341] Call Trace: [10340.217801] [10340.218460] ? show_regs.cold.9+0x22/0x2f [10340.220133] ? watchdog_timer_fn.cold.6+0x62/0x10a [10340.222267] ? __hrtimer_run_queues+0x1c2/0x4d0 [10340.226019] ? watchdog+0x40/0x40 [10340.227571] ? hrtimer_interrupt+0x128/0x2c0 [10340.229289] ? smp_apic_timer_interrupt+0xa6/0x270 [10340.231982] ? apic_timer_interrupt+0xf/0x20 [10340.233994] [10340.234919] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10340.237042] do_raw_spin_lock+0x177/0x190 [10340.239048] _raw_spin_lock+0x21/0x30 [10340.241296] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10340.246025] ? do_raw_spin_unlock+0x75/0x190 [10340.248363] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10340.252600] ? do_wait_intr+0xf0/0xf0 [10340.257990] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10340.269101] mdc_close+0x53b/0xd70 [mdc] [10340.270480] lmv_close+0x18d/0x490 [lmv] [10340.283563] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10340.290496] ll_md_real_close+0xa5/0x2a0 [lustre] [10340.294182] ll_md_close+0x274/0x800 [lustre] [10340.295862] ll_file_release+0x128/0xc50 [lustre] [10340.298792] __fput+0x10d/0x440 [10340.301014] ____fput+0x12/0x20 [10340.302612] task_work_run+0xc6/0x110 [10340.305032] exit_to_usermode_loop+0x1dd/0x1f0 [10340.307035] do_syscall_64+0x3d6/0x440 [10340.309913] entry_SYSCALL_64_after_hwframe+0x49/0xae [10340.312820] RIP: 0033:0x7f1a63983cd8 [10340.314087] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10340.324531] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10340.329145] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10340.337165] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10340.341267] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10340.344616] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10340.350018] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10348.056382] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10348.058176] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10348.070202] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [10348.073240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10348.073329] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10348.073337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10348.073340] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10348.073354] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10348.073358] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10348.073362] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10348.073364] RDX: 0000000000003ad2 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10348.073366] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10348.073368] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10348.073370] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10348.073374] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10348.073376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10348.073378] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10348.073390] Call Trace: [10348.073395] [10348.073401] ? show_regs.cold.9+0x22/0x2f [10348.073411] ? watchdog_timer_fn.cold.6+0x62/0x10a [10348.073417] ? __hrtimer_run_queues+0x1c2/0x4d0 [10348.073421] ? watchdog+0x40/0x40 [10348.073426] ? hrtimer_interrupt+0x128/0x2c0 [10348.073430] ? smp_apic_timer_interrupt+0xa6/0x270 [10348.073443] ? apic_timer_interrupt+0xf/0x20 [10348.073448] [10348.073451] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10348.073455] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10348.073460] do_raw_spin_lock+0x177/0x190 [10348.073464] _raw_spin_lock+0x21/0x30 [10348.073470] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10348.086804] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul [10348.113591] obd_mod_free+0x46/0x140 [ptlrpc] [10348.120339] crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10348.120369] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10348.120376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10348.120506] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10348.120521] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10348.120525] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10348.120530] RAX: 0000000000005191 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10348.120532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10348.120534] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10348.120536] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10348.120538] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10348.120542] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10348.120545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10348.120547] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10348.120624] Call Trace: [10348.120629] [10348.120639] ? show_regs.cold.9+0x22/0x2f [10348.120648] ? watchdog_timer_fn.cold.6+0x62/0x10a [10348.120654] ? __hrtimer_run_queues+0x1c2/0x4d0 [10348.120658] ? watchdog+0x40/0x40 [10348.120663] ? hrtimer_interrupt+0x128/0x2c0 [10348.120667] ? smp_apic_timer_interrupt+0xa6/0x270 [10348.120675] ? apic_timer_interrupt+0xf/0x20 [10348.120680] [10348.120683] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10348.120690] do_raw_spin_lock+0x177/0x190 [10348.120695] _raw_spin_lock+0x21/0x30 [10348.120701] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10348.121117] ? do_raw_spin_unlock+0x75/0x190 [10348.121123] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10348.121350] ? do_wait_intr+0xf0/0xf0 [10348.121364] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10348.128352] mdc_commit_open+0x91/0xb0 [mdc] [10348.128407] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10348.133889] mdc_close+0x53b/0xd70 [mdc] [10348.147013] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10348.152908] lmv_close+0x18d/0x490 [lmv] [10348.159371] after_reply+0xe2c/0x1780 [ptlrpc] [10348.159814] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10348.160005] ? __next_timer_interrupt+0x160/0x160 [10348.160021] ptlrpcd+0xa82/0xe40 [ptlrpc] [10348.160224] ? do_wait_intr+0xf0/0xf0 [10348.160233] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10348.160429] kthread+0x1d1/0x200 [10348.160434] ? set_kthread_struct+0x70/0x70 [10348.160438] ret_from_fork+0x1f/0x30 [10348.469119] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10348.472656] ll_md_real_close+0xa5/0x2a0 [lustre] [10348.476227] ll_md_close+0x274/0x800 [lustre] [10348.478998] ll_file_release+0x128/0xc50 [lustre] [10348.483038] __fput+0x10d/0x440 [10348.484518] ____fput+0x12/0x20 [10348.486127] task_work_run+0xc6/0x110 [10348.488976] exit_to_usermode_loop+0x1dd/0x1f0 [10348.495975] do_syscall_64+0x3d6/0x440 [10348.498222] entry_SYSCALL_64_after_hwframe+0x49/0xae [10348.502195] RIP: 0033:0x7f1a63983cd8 [10348.506142] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10348.526023] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10348.530133] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10348.533640] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10348.536593] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10348.540397] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10348.543511] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10356.053015] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10356.057382] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10356.091315] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10356.110410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10356.119844] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10356.129063] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10356.152251] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10356.160372] RAX: 000000000000580b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10356.171854] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10356.174377] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10356.181318] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10356.188026] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10356.191872] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10356.202153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10356.208910] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10356.217519] Call Trace: [10356.222918] [10356.224892] ? show_regs.cold.9+0x22/0x2f [10356.229206] ? watchdog_timer_fn.cold.6+0x62/0x10a [10356.235284] ? __hrtimer_run_queues+0x1c2/0x4d0 [10356.242716] ? watchdog+0x40/0x40 [10356.244321] ? hrtimer_interrupt+0x128/0x2c0 [10356.246804] ? smp_apic_timer_interrupt+0xa6/0x270 [10356.257302] ? apic_timer_interrupt+0xf/0x20 [10356.258827] [10356.264418] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10356.270259] do_raw_spin_lock+0x177/0x190 [10356.274349] _raw_spin_lock+0x21/0x30 [10356.277440] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10356.285358] ? do_raw_spin_unlock+0x75/0x190 [10356.288417] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10356.298054] ? do_wait_intr+0xf0/0xf0 [10356.299999] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10356.304905] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10356.319268] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10356.323458] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10356.328275] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10356.335316] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10356.337004] mdc_intent_lock+0x353/0x7e0 [mdc] [10356.340389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10356.347301] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10356.350956] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10356.355584] lmv_intent_open+0x338/0x1150 [lmv] [10356.357637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10356.361075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10356.362853] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10356.364220] ll_intent_lock+0xc5/0xaa0 [lustre] [10356.368180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10356.374308] ll_lookup_it+0xe0e/0x2270 [lustre] [10356.377264] ? do_raw_spin_unlock+0x75/0x190 [10356.380703] ? _raw_spin_unlock+0x12/0x30 [10356.382931] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10356.385718] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10356.389146] ll_atomic_open+0x36a/0xd70 [lustre] [10356.393230] lookup_open+0x356/0xb40 [10356.395285] do_last+0x313/0xfc0 [10356.398215] ? nd_jump_root+0xe5/0x160 [10356.400850] ? path_init+0x437/0x520 [10356.402576] path_openat+0xf7/0x500 [10356.404305] ? do_raw_spin_unlock+0x75/0x190 [10356.408689] do_filp_open+0x99/0x140 [10356.410160] ? getname_flags+0x6e/0x330 [10356.411935] ? __check_object_size+0xff/0x256 [10356.416150] ? do_raw_spin_unlock+0x75/0x190 [10356.423090] ? _raw_spin_unlock+0x12/0x30 [10356.427554] do_sys_openat2+0x2b4/0x410 [10356.429715] do_sys_open+0x73/0xa0 [10356.431280] __x64_sys_openat+0x24/0x30 [10356.433453] do_syscall_64+0xc1/0x440 [10356.435391] entry_SYSCALL_64_after_hwframe+0x49/0xae [10356.442896] RIP: 0033:0x7f1a63983332 [10356.444657] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10356.454876] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10356.460126] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10356.463222] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10356.467263] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10356.471221] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10356.475726] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10359.777101] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8336s! [10359.780050] Showing busy workqueues and worker pools: [10359.789306] workqueue events: flags=0x0 [10359.795122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10359.804144] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10359.812995] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10359.821086] pending: sec_gc_main [ptlrpc] [10359.826420] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10359.835527] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10359.851869] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10359.861886] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10359.877891] workqueue events_long: flags=0x0 [10359.879456] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10359.886347] pending: lu_objects_flush [obdclass] [10359.896591] workqueue events_unbound: flags=0x2 [10359.902613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10359.915302] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10359.925208] workqueue events_power_efficient: flags=0x80 [10359.931045] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10359.935041] pending: neigh_periodic_work, do_cache_clean [10359.937668] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10359.941300] pending: neigh_periodic_work, check_lifetime [10359.951116] workqueue mm_percpu_wq: flags=0x8 [10359.955225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10359.961245] pending: vmstat_update [10359.965358] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10359.971389] pending: vmstat_update [10359.981331] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10359.988112] pending: vmstat_update [10359.992756] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10359.999641] pending: vmstat_update [10360.002299] workqueue writeback: flags=0x4a [10360.010400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10360.016380] pending: wb_workfn, wb_workfn [10360.019784] workqueue xprtiod: flags=0xa [10360.021142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10360.023683] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10360.027244] workqueue ipv6_addrconf: flags=0x40008 [10360.028935] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10360.035401] pending: addrconf_verify_work [10360.036699] workqueue knbd0-recv: flags=0x1a [10360.037939] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10360.040327] in-flight: 647:recv_work [10360.042353] workqueue ptlrpc_pinger: flags=0x2 [10360.047351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10360.053729] pending: ptlrpc_pinger_main [ptlrpc] [10360.056952] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10350s workers=2 idle: 652 [10362.636913] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [10362.641104] rcu: 3-...!: (5265763 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [10362.646336] (detected by 1, t=8340230 jiffies, g=269633, q=2988) [10362.652211] Sending NMI from CPU 1 to CPUs 3: [10362.668750] NMI backtrace for cpu 3 [10362.668758] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10362.668760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10362.668762] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10362.668764] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10362.668766] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [10362.668769] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10362.668770] RDX: 0000000000006cfc RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10362.668772] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10362.668773] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10362.668775] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10362.668776] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10362.668777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10362.668779] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10362.668780] Call Trace: [10362.668781] [10362.668782] ? show_regs.cold.9+0x22/0x2f [10362.668783] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10362.668785] ? nmi_cpu_backtrace_handler+0x15/0x40 [10362.668786] ? nmi_handle+0xbd/0x210 [10362.668787] ? default_do_nmi+0x4b/0x1b0 [10362.668788] ? do_nmi+0x280/0x380 [10362.668789] ? end_repeat_nmi+0x16/0x1f [10362.668790] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10362.668791] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10362.668792] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10362.668793] [10362.668794] do_raw_spin_lock+0x177/0x190 [10362.668795] _raw_spin_lock+0x21/0x30 [10362.668796] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10362.668797] obd_mod_free+0x46/0x140 [ptlrpc] [10362.668798] mdc_commit_open+0x91/0xb0 [mdc] [10362.668799] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10362.668800] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10362.668801] after_reply+0xe2c/0x1780 [ptlrpc] [10362.668802] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10362.668803] ? __next_timer_interrupt+0x160/0x160 [10362.668804] ptlrpcd+0xa82/0xe40 [ptlrpc] [10362.668805] ? do_wait_intr+0xf0/0xf0 [10362.668806] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10362.668807] kthread+0x1d1/0x200 [10362.668808] ? set_kthread_struct+0x70/0x70 [10362.668809] ret_from_fork+0x1f/0x30 [10362.668843] rcu: rcu_sched kthread starved for 8339876 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [10362.911460] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10362.915811] rcu: RCU grace-period kthread stack dump: [10362.918478] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10362.921789] Call Trace: [10362.922935] __schedule+0x351/0xcb0 [10362.923885] schedule+0xc0/0x180 [10362.924973] schedule_timeout+0xb4/0x190 [10362.926250] ? do_raw_spin_unlock+0x75/0x190 [10362.928064] ? __next_timer_interrupt+0x160/0x160 [10362.929943] ? prepare_to_swait_event+0xf3/0x140 [10362.931946] rcu_gp_kthread+0x8dd/0x15b0 [10362.933192] ? force_qs_rnp+0x2c0/0x2c0 [10362.934639] kthread+0x1d1/0x200 [10362.935784] ? set_kthread_struct+0x70/0x70 [10362.937241] ret_from_fork+0x1f/0x30 [10362.938569] rcu: Stack dump where RCU GP kthread last ran: [10362.940593] Sending NMI from CPU 1 to CPUs 0: [10362.949699] NMI backtrace for cpu 0 [10362.949707] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10362.949709] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10362.949711] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10362.949713] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10362.949715] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 [10362.949718] RAX: 000000000000759a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10362.949719] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10362.949721] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10362.949722] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10362.949724] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10362.949725] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10362.949727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10362.949728] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10362.949730] Call Trace: [10362.949731] [10362.949732] ? show_regs.cold.9+0x22/0x2f [10362.949733] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10362.949734] ? nmi_cpu_backtrace_handler+0x15/0x40 [10362.949736] ? nmi_handle+0xbd/0x210 [10362.949737] ? default_do_nmi+0x4b/0x1b0 [10362.949738] ? do_nmi+0x280/0x380 [10362.949739] ? end_repeat_nmi+0x16/0x1f [10362.949740] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10362.949741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10362.949742] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10362.949743] [10362.949744] do_raw_spin_lock+0x177/0x190 [10362.949744] _raw_spin_lock+0x21/0x30 [10362.949745] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10362.949747] ? do_raw_spin_unlock+0x75/0x190 [10362.949748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10362.949749] ? do_wait_intr+0xf0/0xf0 [10362.949750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10362.949751] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10362.949752] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10362.949753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10362.949755] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10362.949756] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10362.949757] mdc_intent_lock+0x353/0x7e0 [mdc] [10362.949758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10362.949759] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10362.949760] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10362.949761] lmv_intent_open+0x338/0x1150 [lmv] [10362.949762] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10362.949763] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10362.949765] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10362.949766] ll_intent_lock+0xc5/0xaa0 [lustre] [10362.949768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10362.949769] ll_lookup_it+0xe0e/0x2270 [lustre] [10362.949770] ? do_raw_spin_unlock+0x75/0x190 [10362.949771] ? _raw_spin_unlock+0x12/0x30 [10362.949772] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10362.949773] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10362.949774] ll_atomic_open+0x36a/0xd70 [lustre] [10362.949775] lookup_open+0x356/0xb40 [10362.949777] do_last+0x313/0xfc0 [10362.949778] ? nd_jump_root+0xe5/0x160 [10362.949778] ? path_init+0x437/0x520 [10362.949779] path_openat+0xf7/0x500 [10362.949780] ? do_raw_spin_unlock+0x75/0x190 [10362.949782] do_filp_open+0x99/0x140 [10362.949783] ? getname_flags+0x6e/0x330 [10362.949784] ? __check_object_size+0xff/0x256 [10362.949785] ? do_raw_spin_unlock+0x75/0x190 [10362.949786] ? _raw_spin_unlock+0x12/0x30 [10362.949787] do_sys_openat2+0x2b4/0x410 [10362.949788] do_sys_open+0x73/0xa0 [10362.949789] __x64_sys_openat+0x24/0x30 [10362.949791] do_syscall_64+0xc1/0x440 [10362.949792] entry_SYSCALL_64_after_hwframe+0x49/0xae [10362.949793] RIP: 0033:0x7f1a63983332 [10362.949794] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10362.949797] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10362.949799] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10362.949800] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10362.949802] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10362.949803] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10362.949805] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10368.057768] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10368.062689] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10368.132206] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10368.146159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10368.148716] RIP: 0010:native_safe_halt+0x2e/0x50 [10368.164962] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10368.175637] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10368.179695] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10368.183268] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [10368.186625] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10368.190013] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10368.193285] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10368.196823] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10368.200785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10368.203807] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10368.218841] Call Trace: [10368.220586] [10368.221347] ? show_regs.cold.9+0x22/0x2f [10368.227492] ? watchdog_timer_fn.cold.6+0x62/0x10a [10368.233755] ? __hrtimer_run_queues+0x1c2/0x4d0 [10368.243094] ? watchdog+0x40/0x40 [10368.244259] ? hrtimer_interrupt+0x128/0x2c0 [10368.253668] ? smp_apic_timer_interrupt+0xa6/0x270 [10368.262190] ? apic_timer_interrupt+0xf/0x20 [10368.263748] [10368.270724] ? native_safe_halt+0x2e/0x50 [10368.274078] kvm_wait+0xcd/0xe0 [10368.275043] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10368.277847] do_raw_spin_lock+0x177/0x190 [10368.279637] _raw_spin_lock+0x21/0x30 [10368.280816] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10368.287040] ? do_raw_spin_unlock+0x75/0x190 [10368.288484] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10368.292568] ? do_wait_intr+0xf0/0xf0 [10368.294050] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10368.297778] mdc_close+0x53b/0xd70 [mdc] [10368.299439] lmv_close+0x18d/0x490 [lmv] [10368.300536] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10368.302104] ll_md_real_close+0xa5/0x2a0 [lustre] [10368.306840] ll_md_close+0x274/0x800 [lustre] [10368.308041] ll_file_release+0x128/0xc50 [lustre] [10368.311292] __fput+0x10d/0x440 [10368.312916] ____fput+0x12/0x20 [10368.314469] task_work_run+0xc6/0x110 [10368.316212] exit_to_usermode_loop+0x1dd/0x1f0 [10368.317909] do_syscall_64+0x3d6/0x440 [10368.319718] entry_SYSCALL_64_after_hwframe+0x49/0xae [10368.330376] RIP: 0033:0x7f1a63983cd8 [10368.331683] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10368.352321] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10368.358671] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10368.361985] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10368.364597] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10368.367155] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10368.375133] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10388.053137] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10388.055305] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10388.056083] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10388.063145] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10388.063160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10388.063271] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10388.063277] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10388.063280] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10388.063295] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10388.063300] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10388.063305] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10388.063308] RDX: 0000000000003118 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10388.063310] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10388.063313] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10388.063316] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10388.063319] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10388.063323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10388.063325] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10388.063335] Call Trace: [10388.063341] [10388.063346] ? show_regs.cold.9+0x22/0x2f [10388.063356] ? watchdog_timer_fn.cold.6+0x62/0x10a [10388.063363] ? __hrtimer_run_queues+0x1c2/0x4d0 [10388.063367] ? watchdog+0x40/0x40 [10388.063372] ? hrtimer_interrupt+0x128/0x2c0 [10388.063376] ? smp_apic_timer_interrupt+0xa6/0x270 [10388.063385] ? apic_timer_interrupt+0xf/0x20 [10388.063390] [10388.063393] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10388.063398] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10388.063403] do_raw_spin_lock+0x177/0x190 [10388.063408] _raw_spin_lock+0x21/0x30 [10388.063414] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10388.063702] obd_mod_free+0x46/0x140 [ptlrpc] [10388.065151] mdc_commit_open+0x91/0xb0 [mdc] [10388.065201] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10388.065412] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10388.065623] after_reply+0xe2c/0x1780 [ptlrpc] [10388.065820] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10388.069237] ? __next_timer_interrupt+0x160/0x160 [10388.069253] ptlrpcd+0xa82/0xe40 [ptlrpc] [10388.069504] ? do_wait_intr+0xf0/0xf0 [10388.069513] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10388.072474] kthread+0x1d1/0x200 [10388.072483] ? set_kthread_struct+0x70/0x70 [10388.072488] ret_from_fork+0x1f/0x30 [10388.094034] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10388.094048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10388.094051] RIP: 0010:native_safe_halt+0x2e/0x50 [10388.096693] Modules linked in: [10388.101431] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10388.101444] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10388.101451] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10388.101453] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [10388.101456] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10388.101458] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10388.101460] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10388.101464] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10388.101466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10388.101468] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10388.101477] Call Trace: [10388.101483] [10388.148153] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10388.148650] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10388.148657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10388.148659] RIP: 0010:native_safe_halt+0x2e/0x50 [10388.148672] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10388.148675] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10388.148680] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10388.148682] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10388.148684] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10388.148686] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10388.148688] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10388.148691] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10388.148693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10388.148695] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10388.148703] Call Trace: [10388.148707] [10388.148712] ? show_regs.cold.9+0x22/0x2f [10388.148722] ? watchdog_timer_fn.cold.6+0x62/0x10a [10388.148727] ? __hrtimer_run_queues+0x1c2/0x4d0 [10388.148730] ? watchdog+0x40/0x40 [10388.148734] ? hrtimer_interrupt+0x128/0x2c0 [10388.148738] ? smp_apic_timer_interrupt+0xa6/0x270 [10388.148746] ? apic_timer_interrupt+0xf/0x20 [10388.148750] [10388.148753] ? native_safe_halt+0x2e/0x50 [10388.148757] kvm_wait+0xcd/0xe0 [10388.148764] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10388.148773] do_raw_spin_lock+0x177/0x190 [10388.148778] _raw_spin_lock+0x21/0x30 [10388.148782] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10388.148995] ? do_raw_spin_unlock+0x75/0x190 [10388.149597] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10388.149797] ? do_wait_intr+0xf0/0xf0 [10388.149804] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10388.149985] mdc_close+0x53b/0xd70 [mdc] [10388.150033] lmv_close+0x18d/0x490 [lmv] [10388.150061] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10388.163711] ? show_regs.cold.9+0x22/0x2f [10388.172671] ll_md_real_close+0xa5/0x2a0 [lustre] [10388.172792] ll_md_close+0x274/0x800 [lustre] [10388.172868] ll_file_release+0x128/0xc50 [lustre] [10388.172943] __fput+0x10d/0x440 [10388.172953] ____fput+0x12/0x20 [10388.172958] task_work_run+0xc6/0x110 [10388.172966] exit_to_usermode_loop+0x1dd/0x1f0 [10388.172974] do_syscall_64+0x3d6/0x440 [10388.172978] entry_SYSCALL_64_after_hwframe+0x49/0xae [10388.172986] RIP: 0033:0x7f1a63983cd8 [10388.172992] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10388.172996] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10388.173068] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10388.173071] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10388.173073] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10388.173075] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10388.173078] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10388.826553] ? watchdog_timer_fn.cold.6+0x62/0x10a [10388.831629] ? __hrtimer_run_queues+0x1c2/0x4d0 [10388.834044] ? watchdog+0x40/0x40 [10388.838598] ? hrtimer_interrupt+0x128/0x2c0 [10388.841524] ? smp_apic_timer_interrupt+0xa6/0x270 [10388.844660] ? apic_timer_interrupt+0xf/0x20 [10388.852568] [10388.854381] ? native_safe_halt+0x2e/0x50 [10388.855697] kvm_wait+0xcd/0xe0 [10388.859235] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10388.872510] do_raw_spin_lock+0x177/0x190 [10388.873788] _raw_spin_lock+0x21/0x30 [10388.875047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10388.884908] ? do_raw_spin_unlock+0x75/0x190 [10388.887383] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10388.891654] ? do_wait_intr+0xf0/0xf0 [10388.900581] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10388.906733] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10388.913807] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10388.920574] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10388.928953] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10388.937619] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10388.943050] mdc_intent_lock+0x353/0x7e0 [mdc] [10388.944823] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10388.952515] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10388.955651] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10388.958789] lmv_intent_open+0x338/0x1150 [lmv] [10388.962529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10388.969077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10388.972782] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10388.974601] ll_intent_lock+0xc5/0xaa0 [lustre] [10388.980413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10388.983681] ll_lookup_it+0xe0e/0x2270 [lustre] [10388.991330] ? do_raw_spin_unlock+0x75/0x190 [10388.993504] ? _raw_spin_unlock+0x12/0x30 [10388.995886] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10389.001173] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10389.007604] ll_atomic_open+0x36a/0xd70 [lustre] [10389.010206] lookup_open+0x356/0xb40 [10389.012479] do_last+0x313/0xfc0 [10389.014744] ? nd_jump_root+0xe5/0x160 [10389.015827] ? path_init+0x437/0x520 [10389.017317] path_openat+0xf7/0x500 [10389.022214] ? do_raw_spin_unlock+0x75/0x190 [10389.023299] do_filp_open+0x99/0x140 [10389.025371] ? getname_flags+0x6e/0x330 [10389.026500] ? __check_object_size+0xff/0x256 [10389.028374] ? do_raw_spin_unlock+0x75/0x190 [10389.030110] ? _raw_spin_unlock+0x12/0x30 [10389.031486] do_sys_openat2+0x2b4/0x410 [10389.032881] do_sys_open+0x73/0xa0 [10389.034097] __x64_sys_openat+0x24/0x30 [10389.035766] do_syscall_64+0xc1/0x440 [10389.037323] entry_SYSCALL_64_after_hwframe+0x49/0xae [10389.038904] RIP: 0033:0x7f1a63983332 [10389.039898] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10389.052320] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10389.059578] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10389.064876] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10389.070587] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10389.079776] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10389.095754] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10390.505439] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8367s! [10390.514652] Showing busy workqueues and worker pools: [10390.529084] workqueue events: flags=0x0 [10390.536435] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10390.546618] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10390.553910] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10390.564028] pending: sec_gc_main [ptlrpc] [10390.572938] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10390.599264] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10390.626227] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10390.636296] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10390.648527] workqueue events_long: flags=0x0 [10390.658950] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10390.673754] pending: lu_objects_flush [obdclass] [10390.675822] workqueue events_unbound: flags=0x2 [10390.685369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10390.688761] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10390.694094] workqueue events_power_efficient: flags=0x80 [10390.696870] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10390.700217] pending: neigh_periodic_work, do_cache_clean [10390.704109] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10390.713839] pending: neigh_periodic_work, check_lifetime [10390.716548] workqueue mm_percpu_wq: flags=0x8 [10390.719100] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10390.722726] pending: vmstat_update [10390.724683] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10390.731962] pending: vmstat_update [10390.734105] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10390.741603] pending: vmstat_update [10390.746747] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10390.759295] pending: vmstat_update [10390.760866] workqueue writeback: flags=0x4a [10390.768323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10390.781905] pending: wb_workfn, wb_workfn [10390.784073] workqueue xprtiod: flags=0xa [10390.797744] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10390.805573] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10390.821481] workqueue ipv6_addrconf: flags=0x40008 [10390.823293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10390.832989] pending: addrconf_verify_work [10390.838377] workqueue knbd0-recv: flags=0x1a [10390.840574] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10390.854118] in-flight: 647:recv_work [10390.863586] workqueue ptlrpc_pinger: flags=0x2 [10390.874555] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10390.883091] pending: ptlrpc_pinger_main [ptlrpc] [10390.892637] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10380s workers=2 idle: 652 [10396.057031] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10396.059655] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10396.114607] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10396.124218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10396.133120] RIP: 0010:native_safe_halt+0x2e/0x50 [10396.136179] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10396.156628] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10396.162528] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10396.167077] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [10396.169809] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10396.174244] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10396.178238] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10396.183880] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10396.187683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10396.190654] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10396.197233] Call Trace: [10396.197819] [10396.200266] ? show_regs.cold.9+0x22/0x2f [10396.208072] ? watchdog_timer_fn.cold.6+0x62/0x10a [10396.210916] ? __hrtimer_run_queues+0x1c2/0x4d0 [10396.213863] ? watchdog+0x40/0x40 [10396.218154] ? hrtimer_interrupt+0x128/0x2c0 [10396.220255] ? smp_apic_timer_interrupt+0xa6/0x270 [10396.223379] ? apic_timer_interrupt+0xf/0x20 [10396.225875] [10396.226681] ? native_safe_halt+0x2e/0x50 [10396.230905] kvm_wait+0xcd/0xe0 [10396.232272] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10396.234503] do_raw_spin_lock+0x177/0x190 [10396.242162] _raw_spin_lock+0x21/0x30 [10396.244031] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10396.248320] ? do_raw_spin_unlock+0x75/0x190 [10396.255920] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10396.265259] ? do_wait_intr+0xf0/0xf0 [10396.270394] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10396.274912] mdc_close+0x53b/0xd70 [mdc] [10396.287995] lmv_close+0x18d/0x490 [lmv] [10396.292870] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10396.296642] ll_md_real_close+0xa5/0x2a0 [lustre] [10396.311298] ll_md_close+0x274/0x800 [lustre] [10396.315634] ll_file_release+0x128/0xc50 [lustre] [10396.317443] __fput+0x10d/0x440 [10396.330160] ____fput+0x12/0x20 [10396.332424] task_work_run+0xc6/0x110 [10396.342536] exit_to_usermode_loop+0x1dd/0x1f0 [10396.343930] do_syscall_64+0x3d6/0x440 [10396.351153] entry_SYSCALL_64_after_hwframe+0x49/0xae [10396.359224] RIP: 0033:0x7f1a63983cd8 [10396.365833] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10396.388275] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10396.396223] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10396.412238] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10396.421028] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10396.428623] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10396.435721] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10416.053012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10416.057796] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10416.058007] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10416.058014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10416.058087] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10416.058093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10416.058095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x188/0x6d0 [10416.058110] Code: 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 48 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 <84> c0 75 e6 48 83 05 2c d6 dc 02 01 44 89 f8 f0 66 44 0f b1 33 48 [10416.058114] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10416.058118] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10416.058121] RDX: 0000000000001f17 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10416.058124] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10416.058126] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10416.058129] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10416.058133] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10416.058136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10416.058139] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10416.058147] Call Trace: [10416.058153] [10416.058158] ? show_regs.cold.9+0x22/0x2f [10416.058167] ? watchdog_timer_fn.cold.6+0x62/0x10a [10416.058173] ? __hrtimer_run_queues+0x1c2/0x4d0 [10416.058178] ? watchdog+0x40/0x40 [10416.058182] ? hrtimer_interrupt+0x128/0x2c0 [10416.058186] ? smp_apic_timer_interrupt+0xa6/0x270 [10416.058194] ? apic_timer_interrupt+0xf/0x20 [10416.058199] [10416.058202] ? __pv_queued_spin_lock_slowpath+0x188/0x6d0 [10416.058207] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10416.058212] do_raw_spin_lock+0x177/0x190 [10416.058216] _raw_spin_lock+0x21/0x30 [10416.058222] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10416.058667] obd_mod_free+0x46/0x140 [ptlrpc] [10416.059111] mdc_commit_open+0x91/0xb0 [mdc] [10416.059150] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10416.059368] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10416.059634] after_reply+0xe2c/0x1780 [ptlrpc] [10416.060046] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10416.060608] ? __next_timer_interrupt+0x160/0x160 [10416.060619] ptlrpcd+0xa82/0xe40 [ptlrpc] [10416.061114] ? do_wait_intr+0xf0/0xf0 [10416.061124] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10416.062031] kthread+0x1d1/0x200 [10416.062038] ? set_kthread_struct+0x70/0x70 [10416.062042] ret_from_fork+0x1f/0x30 [10416.066272] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10416.066368] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10416.066381] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10416.066393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10416.066412] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10416.066416] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10416.066425] RAX: 0000000000006bb5 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10416.066428] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10416.066430] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10416.066432] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10416.066435] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10416.066438] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10416.066441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10416.066443] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10416.066455] Call Trace: [10416.066466] [10416.066481] ? show_regs.cold.9+0x22/0x2f [10416.066491] ? watchdog_timer_fn.cold.6+0x62/0x10a [10416.066498] ? __hrtimer_run_queues+0x1c2/0x4d0 [10416.066503] ? watchdog+0x40/0x40 [10416.066507] ? hrtimer_interrupt+0x128/0x2c0 [10416.066511] ? smp_apic_timer_interrupt+0xa6/0x270 [10416.066521] ? apic_timer_interrupt+0xf/0x20 [10416.066526] [10416.066528] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10416.066534] do_raw_spin_lock+0x177/0x190 [10416.066540] _raw_spin_lock+0x21/0x30 [10416.066544] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10416.066823] ? do_raw_spin_unlock+0x75/0x190 [10416.066828] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10416.067275] ? do_wait_intr+0xf0/0xf0 [10416.067285] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10416.067640] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10416.074256] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10416.074723] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10416.074732] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10416.074735] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10416.074749] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10416.074753] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10416.074758] RAX: 0000000000004c88 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10416.074760] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10416.074762] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10416.074764] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10416.074766] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10416.074769] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10416.074772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10416.074774] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10416.074783] Call Trace: [10416.074789] [10416.074795] ? show_regs.cold.9+0x22/0x2f [10416.074804] ? watchdog_timer_fn.cold.6+0x62/0x10a [10416.074810] ? __hrtimer_run_queues+0x1c2/0x4d0 [10416.074814] ? watchdog+0x40/0x40 [10416.074819] ? hrtimer_interrupt+0x128/0x2c0 [10416.074823] ? smp_apic_timer_interrupt+0xa6/0x270 [10416.074833] ? apic_timer_interrupt+0xf/0x20 [10416.074837] [10416.074841] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10416.074847] do_raw_spin_lock+0x177/0x190 [10416.074852] _raw_spin_lock+0x21/0x30 [10416.074857] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10416.076819] ? do_raw_spin_unlock+0x75/0x190 [10416.076829] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10416.078928] ? do_wait_intr+0xf0/0xf0 [10416.078938] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10416.080146] mdc_close+0x53b/0xd70 [mdc] [10416.080187] lmv_close+0x18d/0x490 [lmv] [10416.080214] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10416.091290] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10416.092171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10416.092689] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10416.092730] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10416.092759] mdc_intent_lock+0x353/0x7e0 [mdc] [10416.092786] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10416.092871] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10416.093148] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10416.093183] lmv_intent_open+0x338/0x1150 [lmv] [10416.093213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10416.093313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10416.093397] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10416.093419] ll_intent_lock+0xc5/0xaa0 [lustre] [10416.093816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10416.120310] ll_md_real_close+0xa5/0x2a0 [lustre] [10416.120797] ll_md_close+0x274/0x800 [lustre] [10416.120878] ll_file_release+0x128/0xc50 [lustre] [10416.120977] __fput+0x10d/0x440 [10416.120986] ____fput+0x12/0x20 [10416.120991] task_work_run+0xc6/0x110 [10416.120997] exit_to_usermode_loop+0x1dd/0x1f0 [10416.121009] do_syscall_64+0x3d6/0x440 [10416.121014] entry_SYSCALL_64_after_hwframe+0x49/0xae [10416.121022] RIP: 0033:0x7f1a63983cd8 [10416.121029] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10416.121034] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10416.121039] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10416.121042] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10416.121044] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10416.121050] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10416.121053] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10416.553436] ll_lookup_it+0xe0e/0x2270 [lustre] [10416.555555] ? do_raw_spin_unlock+0x75/0x190 [10416.556865] ? _raw_spin_unlock+0x12/0x30 [10416.558484] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10416.560596] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10416.562794] ll_atomic_open+0x36a/0xd70 [lustre] [10416.565043] lookup_open+0x356/0xb40 [10416.566697] do_last+0x313/0xfc0 [10416.568145] ? nd_jump_root+0xe5/0x160 [10416.569809] ? path_init+0x437/0x520 [10416.571436] path_openat+0xf7/0x500 [10416.572818] ? do_raw_spin_unlock+0x75/0x190 [10416.574690] do_filp_open+0x99/0x140 [10416.576412] ? getname_flags+0x6e/0x330 [10416.578179] ? __check_object_size+0xff/0x256 [10416.580937] ? do_raw_spin_unlock+0x75/0x190 [10416.583337] ? _raw_spin_unlock+0x12/0x30 [10416.587504] do_sys_openat2+0x2b4/0x410 [10416.589609] do_sys_open+0x73/0xa0 [10416.591183] __x64_sys_openat+0x24/0x30 [10416.592973] do_syscall_64+0xc1/0x440 [10416.594596] entry_SYSCALL_64_after_hwframe+0x49/0xae [10416.598181] RIP: 0033:0x7f1a63983332 [10416.600133] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10416.609527] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10416.613427] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10416.616864] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10416.620011] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10416.623455] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10416.626821] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10421.216124] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8398s! [10421.226225] Showing busy workqueues and worker pools: [10421.229927] workqueue events: flags=0x0 [10421.232656] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10421.235836] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10421.242338] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10421.246390] pending: sec_gc_main [ptlrpc] [10421.251735] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10421.259345] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10421.272027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10421.279108] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10421.286225] workqueue events_long: flags=0x0 [10421.289823] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10421.294164] pending: lu_objects_flush [obdclass] [10421.297929] workqueue events_unbound: flags=0x2 [10421.300201] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10421.308977] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10421.317390] workqueue events_power_efficient: flags=0x80 [10421.325407] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10421.334612] pending: neigh_periodic_work, do_cache_clean [10421.339780] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10421.345632] pending: neigh_periodic_work, check_lifetime [10421.348888] workqueue mm_percpu_wq: flags=0x8 [10421.351215] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10421.358699] pending: vmstat_update [10421.365854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10421.371528] pending: vmstat_update [10421.373630] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10421.376566] pending: vmstat_update [10421.378728] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10421.382039] pending: vmstat_update [10421.383690] workqueue writeback: flags=0x4a [10421.385632] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10421.389100] pending: wb_workfn, wb_workfn [10421.391871] workqueue xprtiod: flags=0xa [10421.393726] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10421.397612] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10421.402991] workqueue ipv6_addrconf: flags=0x40008 [10421.406461] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10421.408982] pending: addrconf_verify_work [10421.411509] workqueue knbd0-recv: flags=0x1a [10421.413206] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10421.416020] in-flight: 647:recv_work [10421.417740] workqueue ptlrpc_pinger: flags=0x2 [10421.419734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10421.422495] pending: ptlrpc_pinger_main [ptlrpc] [10421.425314] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10411s workers=2 idle: 652 [10424.057010] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10424.059266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10424.076897] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10424.085201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10424.088505] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10424.090771] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10424.097174] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10424.098953] RAX: 0000000000001311 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10424.101588] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10424.104368] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10424.106765] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10424.109326] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10424.111554] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10424.114261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10424.116145] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10424.121582] Call Trace: [10424.122898] [10424.124261] ? show_regs.cold.9+0x22/0x2f [10424.130421] ? watchdog_timer_fn.cold.6+0x62/0x10a [10424.136425] ? __hrtimer_run_queues+0x1c2/0x4d0 [10424.140432] ? watchdog+0x40/0x40 [10424.141692] ? hrtimer_interrupt+0x128/0x2c0 [10424.148115] ? smp_apic_timer_interrupt+0xa6/0x270 [10424.151342] ? apic_timer_interrupt+0xf/0x20 [10424.154139] [10424.154949] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10424.160567] do_raw_spin_lock+0x177/0x190 [10424.162178] _raw_spin_lock+0x21/0x30 [10424.163573] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10424.167545] ? do_raw_spin_unlock+0x75/0x190 [10424.177612] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10424.185353] ? do_wait_intr+0xf0/0xf0 [10424.188734] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10424.193739] mdc_close+0x53b/0xd70 [mdc] [10424.197282] lmv_close+0x18d/0x490 [lmv] [10424.198719] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10424.201651] ll_md_real_close+0xa5/0x2a0 [lustre] [10424.204144] ll_md_close+0x274/0x800 [lustre] [10424.208953] ll_file_release+0x128/0xc50 [lustre] [10424.212637] __fput+0x10d/0x440 [10424.214368] ____fput+0x12/0x20 [10424.216862] task_work_run+0xc6/0x110 [10424.217975] exit_to_usermode_loop+0x1dd/0x1f0 [10424.222628] do_syscall_64+0x3d6/0x440 [10424.225630] entry_SYSCALL_64_after_hwframe+0x49/0xae [10424.227784] RIP: 0033:0x7f1a63983cd8 [10424.229044] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10424.242130] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10424.248178] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10424.257167] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10424.262969] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10424.268543] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10424.274339] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10444.056082] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10444.058026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10444.067850] Modules linked in: [10444.071162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10444.071247] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10444.071255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10444.071258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10444.071273] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10444.071278] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10444.071282] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10444.071285] RDX: 00000000000045fe RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10444.071288] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10444.071290] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10444.071293] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10444.071296] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10444.071300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10444.071302] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10444.071311] Call Trace: [10444.071316] [10444.071479] ? show_regs.cold.9+0x22/0x2f [10444.071490] ? watchdog_timer_fn.cold.6+0x62/0x10a [10444.071496] ? __hrtimer_run_queues+0x1c2/0x4d0 [10444.071501] ? watchdog+0x40/0x40 [10444.071505] ? hrtimer_interrupt+0x128/0x2c0 [10444.071510] ? smp_apic_timer_interrupt+0xa6/0x270 [10444.071518] ? apic_timer_interrupt+0xf/0x20 [10444.071523] [10444.071526] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10444.071531] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10444.071536] do_raw_spin_lock+0x177/0x190 [10444.071608] _raw_spin_lock+0x21/0x30 [10444.071683] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10444.071971] obd_mod_free+0x46/0x140 [ptlrpc] [10444.072184] mdc_commit_open+0x91/0xb0 [mdc] [10444.072231] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10444.076482] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [10444.096583] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10444.097047] after_reply+0xe2c/0x1780 [ptlrpc] [10444.097283] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10444.097752] ? __next_timer_interrupt+0x160/0x160 [10444.097762] ptlrpcd+0xa82/0xe40 [ptlrpc] [10444.098011] ? do_wait_intr+0xf0/0xf0 [10444.098021] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10444.098474] kthread+0x1d1/0x200 [10444.098481] ? set_kthread_struct+0x70/0x70 [10444.098489] ret_from_fork+0x1f/0x30 [10444.274476] ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10444.291513] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10444.297761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10444.303751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10444.307488] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10444.319332] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10444.322770] RAX: 00000000000020c0 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10444.325814] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10444.329226] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10444.332113] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10444.337251] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10444.344958] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10444.351912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10444.356798] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10444.368552] Call Trace: [10444.370940] [10444.373003] ? show_regs.cold.9+0x22/0x2f [10444.380607] ? watchdog_timer_fn.cold.6+0x62/0x10a [10444.390602] ? __hrtimer_run_queues+0x1c2/0x4d0 [10444.404122] ? watchdog+0x40/0x40 [10444.412378] ? hrtimer_interrupt+0x128/0x2c0 [10444.421423] ? smp_apic_timer_interrupt+0xa6/0x270 [10444.425222] ? apic_timer_interrupt+0xf/0x20 [10444.431979] [10444.432825] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10444.437877] do_raw_spin_lock+0x177/0x190 [10444.443216] _raw_spin_lock+0x21/0x30 [10444.448658] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10444.456232] ? do_raw_spin_unlock+0x75/0x190 [10444.466895] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10444.468700] ? do_wait_intr+0xf0/0xf0 [10444.478188] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10444.484969] mdc_close+0x53b/0xd70 [mdc] [10444.490170] lmv_close+0x18d/0x490 [lmv] [10444.499416] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10444.507210] ll_md_real_close+0xa5/0x2a0 [lustre] [10444.516765] ll_md_close+0x274/0x800 [lustre] [10444.522971] ll_file_release+0x128/0xc50 [lustre] [10444.526797] __fput+0x10d/0x440 [10444.532091] ____fput+0x12/0x20 [10444.534237] task_work_run+0xc6/0x110 [10444.538396] exit_to_usermode_loop+0x1dd/0x1f0 [10444.540498] do_syscall_64+0x3d6/0x440 [10444.545196] entry_SYSCALL_64_after_hwframe+0x49/0xae [10444.551028] RIP: 0033:0x7f1a63983cd8 [10444.559444] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10444.599139] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10444.611992] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10444.623829] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10444.635845] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10444.640840] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10444.649354] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10448.053409] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10448.057917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10448.122187] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10448.141552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10448.153362] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10448.155115] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10448.176278] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10448.181101] RAX: 000000000000392b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10448.184403] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10448.190064] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10448.192094] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10448.197523] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10448.200725] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10448.205153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10448.212111] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10448.217086] Call Trace: [10448.218763] [10448.219303] ? show_regs.cold.9+0x22/0x2f [10448.221934] ? watchdog_timer_fn.cold.6+0x62/0x10a [10448.224985] ? __hrtimer_run_queues+0x1c2/0x4d0 [10448.228922] ? watchdog+0x40/0x40 [10448.231621] ? hrtimer_interrupt+0x128/0x2c0 [10448.235546] ? smp_apic_timer_interrupt+0xa6/0x270 [10448.242399] ? apic_timer_interrupt+0xf/0x20 [10448.244499] [10448.247003] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10448.249121] do_raw_spin_lock+0x177/0x190 [10448.252780] _raw_spin_lock+0x21/0x30 [10448.254144] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10448.257424] ? do_raw_spin_unlock+0x75/0x190 [10448.258630] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10448.262112] ? do_wait_intr+0xf0/0xf0 [10448.263359] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10448.266037] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10448.270388] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10448.276184] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10448.282292] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10448.284468] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10448.294108] mdc_intent_lock+0x353/0x7e0 [mdc] [10448.299955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10448.311385] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10448.330425] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10448.344571] lmv_intent_open+0x338/0x1150 [lmv] [10448.355856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10448.370909] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10448.388957] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10448.392611] ll_intent_lock+0xc5/0xaa0 [lustre] [10448.397417] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10448.402292] ll_lookup_it+0xe0e/0x2270 [lustre] [10448.407607] ? do_raw_spin_unlock+0x75/0x190 [10448.410199] ? _raw_spin_unlock+0x12/0x30 [10448.417568] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10448.421819] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10448.427999] ll_atomic_open+0x36a/0xd70 [lustre] [10448.430325] lookup_open+0x356/0xb40 [10448.433561] do_last+0x313/0xfc0 [10448.437049] ? nd_jump_root+0xe5/0x160 [10448.442911] ? path_init+0x437/0x520 [10448.447071] path_openat+0xf7/0x500 [10448.449023] ? do_raw_spin_unlock+0x75/0x190 [10448.450514] do_filp_open+0x99/0x140 [10448.457139] ? getname_flags+0x6e/0x330 [10448.459328] ? __check_object_size+0xff/0x256 [10448.467083] ? do_raw_spin_unlock+0x75/0x190 [10448.468492] ? _raw_spin_unlock+0x12/0x30 [10448.471079] do_sys_openat2+0x2b4/0x410 [10448.473174] do_sys_open+0x73/0xa0 [10448.481927] __x64_sys_openat+0x24/0x30 [10448.487381] do_syscall_64+0xc1/0x440 [10448.491781] entry_SYSCALL_64_after_hwframe+0x49/0xae [10448.497337] RIP: 0033:0x7f1a63983332 [10448.498379] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10448.529072] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10448.532368] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10448.542921] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10448.547931] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10448.555238] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10448.562946] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10451.943138] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8428s! [10451.959422] Showing busy workqueues and worker pools: [10451.963874] workqueue events: flags=0x0 [10451.977596] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10451.980424] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10451.989067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10451.991346] pending: sec_gc_main [ptlrpc] [10451.993974] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10451.997538] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10452.013325] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10452.025361] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10452.036928] workqueue events_long: flags=0x0 [10452.043052] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10452.047613] pending: lu_objects_flush [obdclass] [10452.054374] workqueue events_unbound: flags=0x2 [10452.057014] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10452.058123] pwq 8: [10452.061417] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10452.061499] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10452.061506] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10452.061509] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10452.061522] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10452.061527] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10452.061531] RAX: 000000000000633a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10452.061533] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10452.061536] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10452.061538] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10452.061541] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10452.061544] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10452.061547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10452.061560] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10452.061568] Call Trace: [10452.061574] [10452.061581] ? show_regs.cold.9+0x22/0x2f [10452.061590] ? watchdog_timer_fn.cold.6+0x62/0x10a [10452.061597] ? __hrtimer_run_queues+0x1c2/0x4d0 [10452.061601] ? watchdog+0x40/0x40 [10452.061605] ? hrtimer_interrupt+0x128/0x2c0 [10452.061610] ? smp_apic_timer_interrupt+0xa6/0x270 [10452.061618] ? apic_timer_interrupt+0xf/0x20 [10452.061622] [10452.061625] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10452.061630] do_raw_spin_lock+0x177/0x190 [10452.061634] _raw_spin_lock+0x21/0x30 [10452.061639] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10452.062069] ? do_raw_spin_unlock+0x75/0x190 [10452.062075] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10452.062347] ? do_wait_intr+0xf0/0xf0 [10452.062356] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10452.062637] mdc_close+0x53b/0xd70 [mdc] [10452.062677] lmv_close+0x18d/0x490 [lmv] [10452.062709] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10452.062957] ll_md_real_close+0xa5/0x2a0 [lustre] [10452.063042] ll_md_close+0x274/0x800 [lustre] [10452.063129] ll_file_release+0x128/0xc50 [lustre] [10452.063386] __fput+0x10d/0x440 [10452.063395] ____fput+0x12/0x20 [10452.063399] task_work_run+0xc6/0x110 [10452.063407] exit_to_usermode_loop+0x1dd/0x1f0 [10452.063414] do_syscall_64+0x3d6/0x440 [10452.063418] entry_SYSCALL_64_after_hwframe+0x49/0xae [10452.063424] RIP: 0033:0x7f1a63983cd8 [10452.063431] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10452.063435] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10452.063440] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10452.063442] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10452.063445] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10452.063446] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10452.063449] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10452.613185] cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10452.620786] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10452.631054] workqueue events_power_efficient: flags=0x80 [10452.638390] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10452.651365] pending: neigh_periodic_work, do_cache_clean [10452.653514] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10452.662757] pending: neigh_periodic_work, check_lifetime [10452.675821] workqueue mm_percpu_wq: flags=0x8 [10452.680355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10452.691779] pending: vmstat_update [10452.698478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10452.707967] pending: vmstat_update [10452.716063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10452.725490] pending: vmstat_update [10452.732103] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10452.740477] pending: vmstat_update [10452.742027] workqueue writeback: flags=0x4a [10452.748840] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10452.757292] pending: wb_workfn, wb_workfn [10452.767274] workqueue xprtiod: flags=0xa [10452.775778] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10452.779665] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10452.791059] workqueue ipv6_addrconf: flags=0x40008 [10452.802763] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10452.813698] pending: addrconf_verify_work [10452.818787] workqueue knbd0-recv: flags=0x1a [10452.823954] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10452.832582] in-flight: 647:recv_work [10452.834553] workqueue ptlrpc_pinger: flags=0x2 [10452.846340] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10452.855000] pending: ptlrpc_pinger_main [ptlrpc] [10452.863969] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10442s workers=2 idle: 652 [10472.060281] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10472.060297] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10472.067905] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10472.067986] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10472.067994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10472.067998] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10472.068012] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10472.068017] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10472.068021] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10472.068024] RDX: 00000000000046d7 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10472.068026] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10472.068028] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10472.068031] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10472.068034] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10472.068037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10472.068039] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10472.068047] Call Trace: [10472.068052] [10472.068058] ? show_regs.cold.9+0x22/0x2f [10472.068068] ? watchdog_timer_fn.cold.6+0x62/0x10a [10472.068075] ? __hrtimer_run_queues+0x1c2/0x4d0 [10472.068079] ? watchdog+0x40/0x40 [10472.068083] ? hrtimer_interrupt+0x128/0x2c0 [10472.068087] ? smp_apic_timer_interrupt+0xa6/0x270 [10472.068095] ? apic_timer_interrupt+0xf/0x20 [10472.068100] [10472.068103] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10472.068108] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10472.068114] do_raw_spin_lock+0x177/0x190 [10472.068119] _raw_spin_lock+0x21/0x30 [10472.068125] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10472.070589] obd_mod_free+0x46/0x140 [ptlrpc] [10472.070791] mdc_commit_open+0x91/0xb0 [mdc] [10472.070831] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10472.071022] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10472.072314] after_reply+0xe2c/0x1780 [ptlrpc] [10472.072545] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10472.072809] ? __next_timer_interrupt+0x160/0x160 [10472.072819] ptlrpcd+0xa82/0xe40 [ptlrpc] [10472.073020] ? do_wait_intr+0xf0/0xf0 [10472.073029] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10472.073439] kthread+0x1d1/0x200 [10472.073445] ? set_kthread_struct+0x70/0x70 [10472.073449] ret_from_fork+0x1f/0x30 [10472.488278] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10472.571897] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10472.585692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10472.613172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10472.622874] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10472.656394] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10472.665795] RAX: 0000000000007310 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10472.683225] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10472.699691] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10472.715562] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10472.725281] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10472.736100] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10472.750144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10472.755856] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10472.768729] Call Trace: [10472.773843] [10472.778134] ? show_regs.cold.9+0x22/0x2f [10472.782814] ? watchdog_timer_fn.cold.6+0x62/0x10a [10472.787857] ? __hrtimer_run_queues+0x1c2/0x4d0 [10472.795096] ? watchdog+0x40/0x40 [10472.796421] ? hrtimer_interrupt+0x128/0x2c0 [10472.812436] ? smp_apic_timer_interrupt+0xa6/0x270 [10472.819741] ? apic_timer_interrupt+0xf/0x20 [10472.826141] [10472.828910] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10472.835053] do_raw_spin_lock+0x177/0x190 [10472.838535] _raw_spin_lock+0x21/0x30 [10472.842690] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10472.854638] ? do_raw_spin_unlock+0x75/0x190 [10472.856729] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10472.860780] ? do_wait_intr+0xf0/0xf0 [10472.869646] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10472.880866] mdc_close+0x53b/0xd70 [mdc] [10472.887588] lmv_close+0x18d/0x490 [lmv] [10472.894631] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10472.904471] ll_md_real_close+0xa5/0x2a0 [lustre] [10472.908625] ll_md_close+0x274/0x800 [lustre] [10472.915712] ll_file_release+0x128/0xc50 [lustre] [10472.924283] __fput+0x10d/0x440 [10472.933078] ____fput+0x12/0x20 [10472.936534] task_work_run+0xc6/0x110 [10472.937943] exit_to_usermode_loop+0x1dd/0x1f0 [10472.951920] do_syscall_64+0x3d6/0x440 [10472.963423] entry_SYSCALL_64_after_hwframe+0x49/0xae [10472.973929] RIP: 0033:0x7f1a63983cd8 [10472.978959] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10473.013613] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10473.024690] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10473.034962] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10473.048806] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10473.060229] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10473.074357] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10480.054759] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [10480.059234] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10480.061459] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10480.061533] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10480.061541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10480.061544] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10480.061902] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10480.061907] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10480.061912] RAX: 0000000000006787 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10480.061914] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10480.061917] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10480.061919] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10480.061921] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10480.061925] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10480.061928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10480.061931] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10480.061941] Call Trace: [10480.061947] [10480.061952] ? show_regs.cold.9+0x22/0x2f [10480.061976] ? watchdog_timer_fn.cold.6+0x62/0x10a [10480.061982] ? __hrtimer_run_queues+0x1c2/0x4d0 [10480.061986] ? watchdog+0x40/0x40 [10480.061991] ? hrtimer_interrupt+0x128/0x2c0 [10480.061995] ? smp_apic_timer_interrupt+0xa6/0x270 [10480.062004] ? apic_timer_interrupt+0xf/0x20 [10480.062009] [10480.062012] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10480.062017] do_raw_spin_lock+0x177/0x190 [10480.062022] _raw_spin_lock+0x21/0x30 [10480.062027] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10480.062871] ? do_raw_spin_unlock+0x75/0x190 [10480.062878] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10480.063094] ? do_wait_intr+0xf0/0xf0 [10480.063103] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10480.064808] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10480.075691] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10480.075780] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10480.075975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10480.075978] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10480.075991] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10480.075994] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10480.075999] RAX: 0000000000006323 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10480.076001] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10480.076003] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10480.076005] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10480.076007] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10480.076010] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10480.076012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10480.076014] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10480.076022] Call Trace: [10480.076026] [10480.076031] ? show_regs.cold.9+0x22/0x2f [10480.076039] ? watchdog_timer_fn.cold.6+0x62/0x10a [10480.076044] ? __hrtimer_run_queues+0x1c2/0x4d0 [10480.076048] ? watchdog+0x40/0x40 [10480.076051] ? hrtimer_interrupt+0x128/0x2c0 [10480.076054] ? smp_apic_timer_interrupt+0xa6/0x270 [10480.076061] ? apic_timer_interrupt+0xf/0x20 [10480.076065] [10480.076068] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10480.076073] do_raw_spin_lock+0x177/0x190 [10480.076076] _raw_spin_lock+0x21/0x30 [10480.076080] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10480.077321] ? do_raw_spin_unlock+0x75/0x190 [10480.077329] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10480.079709] ? do_wait_intr+0xf0/0xf0 [10480.079720] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10480.080712] mdc_close+0x53b/0xd70 [mdc] [10480.080752] lmv_close+0x18d/0x490 [lmv] [10480.080782] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10480.082364] ll_md_real_close+0xa5/0x2a0 [lustre] [10480.082446] ll_md_close+0x274/0x800 [lustre] [10480.082528] ll_file_release+0x128/0xc50 [lustre] [10480.082613] __fput+0x10d/0x440 [10480.082622] ____fput+0x12/0x20 [10480.082627] task_work_run+0xc6/0x110 [10480.082635] exit_to_usermode_loop+0x1dd/0x1f0 [10480.082643] do_syscall_64+0x3d6/0x440 [10480.082648] entry_SYSCALL_64_after_hwframe+0x49/0xae [10480.082657] RIP: 0033:0x7f1a63983cd8 [10480.082666] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10480.082670] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10480.082675] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10480.082677] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10480.082680] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10480.082682] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10480.082685] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10480.383829] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10480.387196] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10480.392890] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10480.395890] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10480.398388] mdc_intent_lock+0x353/0x7e0 [mdc] [10480.400515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10480.403750] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10480.406956] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10480.410049] lmv_intent_open+0x338/0x1150 [lmv] [10480.413284] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10480.422125] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10480.432248] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10480.440080] ll_intent_lock+0xc5/0xaa0 [lustre] [10480.446263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10480.451385] ll_lookup_it+0xe0e/0x2270 [lustre] [10480.454126] ? do_raw_spin_unlock+0x75/0x190 [10480.456617] ? _raw_spin_unlock+0x12/0x30 [10480.458256] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10480.461232] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10480.464670] ll_atomic_open+0x36a/0xd70 [lustre] [10480.467507] lookup_open+0x356/0xb40 [10480.469249] do_last+0x313/0xfc0 [10480.471243] ? nd_jump_root+0xe5/0x160 [10480.473288] ? path_init+0x437/0x520 [10480.475403] path_openat+0xf7/0x500 [10480.477430] ? do_raw_spin_unlock+0x75/0x190 [10480.479747] do_filp_open+0x99/0x140 [10480.481190] ? getname_flags+0x6e/0x330 [10480.484758] ? __check_object_size+0xff/0x256 [10480.487629] ? do_raw_spin_unlock+0x75/0x190 [10480.490388] ? _raw_spin_unlock+0x12/0x30 [10480.493222] do_sys_openat2+0x2b4/0x410 [10480.495562] do_sys_open+0x73/0xa0 [10480.498021] __x64_sys_openat+0x24/0x30 [10480.500052] do_syscall_64+0xc1/0x440 [10480.502437] entry_SYSCALL_64_after_hwframe+0x49/0xae [10480.505189] RIP: 0033:0x7f1a63983332 [10480.507642] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10480.519503] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10480.524440] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10480.528682] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10480.533491] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10480.537137] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10480.541928] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10483.172241] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8460s! [10483.177178] Showing busy workqueues and worker pools: [10483.181053] workqueue events: flags=0x0 [10483.184667] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10483.189165] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10483.195701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10483.200526] pending: sec_gc_main [ptlrpc] [10483.203616] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10483.207777] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10483.214502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10483.218097] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10483.224342] workqueue events_long: flags=0x0 [10483.226658] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10483.230904] pending: lu_objects_flush [obdclass] [10483.234108] workqueue events_unbound: flags=0x2 [10483.236918] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10483.240486] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10483.246922] workqueue events_power_efficient: flags=0x80 [10483.251143] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10483.255171] pending: neigh_periodic_work, do_cache_clean [10483.258884] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10483.263391] pending: neigh_periodic_work, check_lifetime [10483.267619] workqueue mm_percpu_wq: flags=0x8 [10483.269823] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10483.274570] pending: vmstat_update [10483.277947] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10483.282468] pending: vmstat_update [10483.284209] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10483.288201] pending: vmstat_update [10483.290252] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10483.294826] pending: vmstat_update [10483.297400] workqueue writeback: flags=0x4a [10483.301189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10483.305457] pending: wb_workfn, wb_workfn [10483.307842] workqueue xprtiod: flags=0xa [10483.310015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10483.313811] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10483.320136] workqueue ipv6_addrconf: flags=0x40008 [10483.323228] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10483.327515] pending: addrconf_verify_work [10483.329735] workqueue knbd0-recv: flags=0x1a [10483.331855] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10483.336494] in-flight: 647:recv_work [10483.339931] workqueue ptlrpc_pinger: flags=0x2 [10483.342175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10483.345924] pending: ptlrpc_pinger_main [ptlrpc] [10483.349382] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10473s workers=2 idle: 652 [10500.061947] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10500.063235] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10500.068990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10500.069496] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10500.069504] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10500.069507] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10500.069644] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10500.069649] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10500.069654] RAX: 0000000000007e0a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10500.069657] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10500.069659] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10500.069662] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10500.069664] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10500.069668] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10500.069671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10500.069673] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10500.069681] Call Trace: [10500.069686] [10500.069692] ? show_regs.cold.9+0x22/0x2f [10500.069702] ? watchdog_timer_fn.cold.6+0x62/0x10a [10500.069708] ? __hrtimer_run_queues+0x1c2/0x4d0 [10500.069713] ? watchdog+0x40/0x40 [10500.069718] ? hrtimer_interrupt+0x128/0x2c0 [10500.069722] ? smp_apic_timer_interrupt+0xa6/0x270 [10500.069733] ? apic_timer_interrupt+0xf/0x20 [10500.069738] [10500.069740] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10500.069747] do_raw_spin_lock+0x177/0x190 [10500.069753] _raw_spin_lock+0x21/0x30 [10500.069759] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10500.070075] ? do_raw_spin_unlock+0x75/0x190 [10500.070081] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10500.070430] ? do_wait_intr+0xf0/0xf0 [10500.070438] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10500.070638] mdc_close+0x53b/0xd70 [mdc] [10500.070675] lmv_close+0x18d/0x490 [lmv] [10500.070705] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10500.070810] ll_md_real_close+0xa5/0x2a0 [lustre] [10500.070889] ll_md_close+0x274/0x800 [lustre] [10500.070970] ll_file_release+0x128/0xc50 [lustre] [10500.071617] __fput+0x10d/0x440 [10500.071632] ____fput+0x12/0x20 [10500.071637] task_work_run+0xc6/0x110 [10500.071645] exit_to_usermode_loop+0x1dd/0x1f0 [10500.071652] do_syscall_64+0x3d6/0x440 [10500.071657] entry_SYSCALL_64_after_hwframe+0x49/0xae [10500.071664] RIP: 0033:0x7f1a63983cd8 [10500.071671] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10500.071675] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10500.071679] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10500.071682] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10500.071684] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10500.071686] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10500.071688] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10500.483835] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10500.518938] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10500.524950] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10500.531142] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10500.534287] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10500.545017] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10500.553749] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10500.557215] RDX: 0000000000002739 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10500.562135] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10500.565894] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10500.569274] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10500.572929] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10500.577019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10500.579879] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10500.583532] Call Trace: [10500.584744] [10500.585695] ? show_regs.cold.9+0x22/0x2f [10500.587445] ? watchdog_timer_fn.cold.6+0x62/0x10a [10500.592622] ? __hrtimer_run_queues+0x1c2/0x4d0 [10500.594933] ? watchdog+0x40/0x40 [10500.596911] ? hrtimer_interrupt+0x128/0x2c0 [10500.599052] ? smp_apic_timer_interrupt+0xa6/0x270 [10500.601458] ? apic_timer_interrupt+0xf/0x20 [10500.603814] [10500.604855] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10500.607707] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10500.610278] do_raw_spin_lock+0x177/0x190 [10500.612020] _raw_spin_lock+0x21/0x30 [10500.613757] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10500.616545] obd_mod_free+0x46/0x140 [ptlrpc] [10500.618934] mdc_commit_open+0x91/0xb0 [mdc] [10500.621011] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10500.623775] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10500.626530] after_reply+0xe2c/0x1780 [ptlrpc] [10500.628950] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10500.631754] ? __next_timer_interrupt+0x160/0x160 [10500.634286] ptlrpcd+0xa82/0xe40 [ptlrpc] [10500.636655] ? do_wait_intr+0xf0/0xf0 [10500.638440] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10500.645398] kthread+0x1d1/0x200 [10500.646583] ? set_kthread_struct+0x70/0x70 [10500.647990] ret_from_fork+0x1f/0x30 [10508.058382] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10508.064512] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10508.073043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10508.073455] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10508.073462] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10508.073464] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10508.073478] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10508.073482] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10508.073486] RAX: 0000000000006f14 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10508.073489] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10508.073491] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10508.073493] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10508.073495] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10508.073498] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10508.073501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10508.073503] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10508.073512] Call Trace: [10508.073518] [10508.073524] ? show_regs.cold.9+0x22/0x2f [10508.073533] ? watchdog_timer_fn.cold.6+0x62/0x10a [10508.073539] ? __hrtimer_run_queues+0x1c2/0x4d0 [10508.073543] ? watchdog+0x40/0x40 [10508.073547] ? hrtimer_interrupt+0x128/0x2c0 [10508.073551] ? smp_apic_timer_interrupt+0xa6/0x270 [10508.073558] ? apic_timer_interrupt+0xf/0x20 [10508.073562] [10508.073565] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10508.073570] do_raw_spin_lock+0x177/0x190 [10508.073575] _raw_spin_lock+0x21/0x30 [10508.073579] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10508.074234] ? do_raw_spin_unlock+0x75/0x190 [10508.074240] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10508.074452] ? do_wait_intr+0xf0/0xf0 [10508.074461] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10508.074663] mdc_close+0x53b/0xd70 [mdc] [10508.074723] lmv_close+0x18d/0x490 [lmv] [10508.074760] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10508.078670] ll_md_real_close+0xa5/0x2a0 [lustre] [10508.079195] ll_md_close+0x274/0x800 [lustre] [10508.079278] ll_file_release+0x128/0xc50 [lustre] [10508.079365] __fput+0x10d/0x440 [10508.079375] ____fput+0x12/0x20 [10508.079379] task_work_run+0xc6/0x110 [10508.079387] exit_to_usermode_loop+0x1dd/0x1f0 [10508.079394] do_syscall_64+0x3d6/0x440 [10508.079398] entry_SYSCALL_64_after_hwframe+0x49/0xae [10508.079408] RIP: 0033:0x7f1a63983cd8 [10508.079416] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10508.079419] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10508.079425] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10508.079427] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10508.079430] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10508.079432] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10508.079434] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10508.466850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10508.490375] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10508.496308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10508.499519] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10508.502506] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10508.514336] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10508.518325] RAX: 0000000000003d89 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10508.521578] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10508.524951] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10508.528774] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10508.532142] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10508.536844] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10508.545459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10508.551347] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10508.559037] Call Trace: [10508.559764] [10508.560740] ? show_regs.cold.9+0x22/0x2f [10508.564471] ? watchdog_timer_fn.cold.6+0x62/0x10a [10508.567368] ? __hrtimer_run_queues+0x1c2/0x4d0 [10508.572048] ? watchdog+0x40/0x40 [10508.575575] ? hrtimer_interrupt+0x128/0x2c0 [10508.580326] ? smp_apic_timer_interrupt+0xa6/0x270 [10508.583652] ? apic_timer_interrupt+0xf/0x20 [10508.586982] [10508.588103] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10508.591098] do_raw_spin_lock+0x177/0x190 [10508.593581] _raw_spin_lock+0x21/0x30 [10508.595956] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10508.600449] ? do_raw_spin_unlock+0x75/0x190 [10508.602668] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10508.605730] ? do_wait_intr+0xf0/0xf0 [10508.609953] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10508.618620] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10508.626308] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10508.630614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10508.633779] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10508.637659] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10508.640716] mdc_intent_lock+0x353/0x7e0 [mdc] [10508.644352] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10508.647396] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10508.652876] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10508.661659] lmv_intent_open+0x338/0x1150 [lmv] [10508.663887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10508.668144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10508.689363] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10508.700598] ll_intent_lock+0xc5/0xaa0 [lustre] [10508.714101] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10508.723311] ll_lookup_it+0xe0e/0x2270 [lustre] [10508.730882] ? do_raw_spin_unlock+0x75/0x190 [10508.735370] ? _raw_spin_unlock+0x12/0x30 [10508.739341] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10508.748427] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10508.755391] ll_atomic_open+0x36a/0xd70 [lustre] [10508.762921] lookup_open+0x356/0xb40 [10508.775328] do_last+0x313/0xfc0 [10508.776333] ? nd_jump_root+0xe5/0x160 [10508.786626] ? path_init+0x437/0x520 [10508.793920] path_openat+0xf7/0x500 [10508.805222] ? do_raw_spin_unlock+0x75/0x190 [10508.814779] do_filp_open+0x99/0x140 [10508.825659] ? getname_flags+0x6e/0x330 [10508.843425] ? __check_object_size+0xff/0x256 [10508.856712] ? do_raw_spin_unlock+0x75/0x190 [10508.866420] ? _raw_spin_unlock+0x12/0x30 [10508.867725] do_sys_openat2+0x2b4/0x410 [10508.868855] do_sys_open+0x73/0xa0 [10508.880163] __x64_sys_openat+0x24/0x30 [10508.887733] do_syscall_64+0xc1/0x440 [10508.889668] entry_SYSCALL_64_after_hwframe+0x49/0xae [10508.904537] RIP: 0033:0x7f1a63983332 [10508.909858] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10508.941919] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10508.955211] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10508.974843] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10508.992757] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10509.018111] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10509.043491] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10513.376230] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8490s! [10513.382665] Showing busy workqueues and worker pools: [10513.391685] workqueue events: flags=0x0 [10513.397430] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10513.403450] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10513.412386] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10513.419201] pending: sec_gc_main [ptlrpc] [10513.427460] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10513.434192] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10513.443895] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10513.448543] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10513.469241] workqueue events_long: flags=0x0 [10513.471592] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10513.480689] pending: lu_objects_flush [obdclass] [10513.483899] workqueue events_unbound: flags=0x2 [10513.485648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10513.494865] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10513.510227] workqueue events_power_efficient: flags=0x80 [10513.514436] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10513.525702] pending: neigh_periodic_work, do_cache_clean [10513.531603] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10513.537472] pending: neigh_periodic_work, check_lifetime [10513.540428] workqueue mm_percpu_wq: flags=0x8 [10513.542588] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10513.546656] pending: vmstat_update [10513.548871] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10513.552642] pending: vmstat_update [10513.554594] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10513.560279] pending: vmstat_update [10513.562770] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10513.566759] pending: vmstat_update [10513.569612] workqueue writeback: flags=0x4a [10513.572294] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10513.576332] pending: wb_workfn, wb_workfn [10513.580813] workqueue xprtiod: flags=0xa [10513.583544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10513.586855] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10513.592347] workqueue ipv6_addrconf: flags=0x40008 [10513.594639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10513.602456] pending: addrconf_verify_work [10513.605578] workqueue knbd0-recv: flags=0x1a [10513.610512] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10513.621227] in-flight: 647:recv_work [10513.623637] workqueue ptlrpc_pinger: flags=0x2 [10513.627330] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10513.630850] pending: ptlrpc_pinger_main [ptlrpc] [10513.634411] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10503s workers=2 idle: 652 [10528.061967] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [10528.063181] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [10528.064792] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10528.064882] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10528.064889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10528.064892] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10528.064907] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10528.064911] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10528.064916] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10528.064918] RDX: 0000000000003369 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10528.064921] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10528.064923] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10528.064925] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10528.064929] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10528.064931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10528.064934] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10528.064943] Call Trace: [10528.064948] [10528.064953] ? show_regs.cold.9+0x22/0x2f [10528.064962] ? watchdog_timer_fn.cold.6+0x62/0x10a [10528.064969] ? __hrtimer_run_queues+0x1c2/0x4d0 [10528.064973] ? watchdog+0x40/0x40 [10528.064977] ? hrtimer_interrupt+0x128/0x2c0 [10528.064981] ? smp_apic_timer_interrupt+0xa6/0x270 [10528.064989] ? apic_timer_interrupt+0xf/0x20 [10528.064994] [10528.064996] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10528.065001] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10528.065006] do_raw_spin_lock+0x177/0x190 [10528.065011] _raw_spin_lock+0x21/0x30 [10528.068563] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10528.069131] obd_mod_free+0x46/0x140 [ptlrpc] [10528.070539] mdc_commit_open+0x91/0xb0 [mdc] [10528.070574] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10528.071133] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10528.074966] after_reply+0xe2c/0x1780 [ptlrpc] [10528.075228] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10528.075433] ? __next_timer_interrupt+0x160/0x160 [10528.075443] ptlrpcd+0xa82/0xe40 [ptlrpc] [10528.075641] ? do_wait_intr+0xf0/0xf0 [10528.075651] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10528.085630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [10528.159654] kthread+0x1d1/0x200 [10528.159686] ? set_kthread_struct+0x70/0x70 [10528.159692] ret_from_fork+0x1f/0x30 [10528.602067] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10528.677186] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10528.689979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10528.700504] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10528.702505] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10528.721097] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10528.728684] RAX: 000000000000788d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10528.731198] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10528.740765] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10528.745152] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10528.753291] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10528.765500] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10528.773087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10528.784778] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10528.791665] Call Trace: [10528.795322] [10528.799355] ? show_regs.cold.9+0x22/0x2f [10528.804571] ? watchdog_timer_fn.cold.6+0x62/0x10a [10528.811938] ? __hrtimer_run_queues+0x1c2/0x4d0 [10528.814429] ? watchdog+0x40/0x40 [10528.816402] ? hrtimer_interrupt+0x128/0x2c0 [10528.818733] ? smp_apic_timer_interrupt+0xa6/0x270 [10528.821327] ? apic_timer_interrupt+0xf/0x20 [10528.823627] [10528.825330] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10528.828547] do_raw_spin_lock+0x177/0x190 [10528.830591] _raw_spin_lock+0x21/0x30 [10528.832419] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10528.835205] ? do_raw_spin_unlock+0x75/0x190 [10528.836914] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10528.839508] ? do_wait_intr+0xf0/0xf0 [10528.841117] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10528.843725] mdc_close+0x53b/0xd70 [mdc] [10528.845637] lmv_close+0x18d/0x490 [lmv] [10528.847269] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10528.849616] ll_md_real_close+0xa5/0x2a0 [lustre] [10528.851372] ll_md_close+0x274/0x800 [lustre] [10528.853232] ll_file_release+0x128/0xc50 [lustre] [10528.855559] __fput+0x10d/0x440 [10528.857032] ____fput+0x12/0x20 [10528.858224] task_work_run+0xc6/0x110 [10528.860151] exit_to_usermode_loop+0x1dd/0x1f0 [10528.862014] do_syscall_64+0x3d6/0x440 [10528.863432] entry_SYSCALL_64_after_hwframe+0x49/0xae [10528.865836] RIP: 0033:0x7f1a63983cd8 [10528.867743] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10528.875204] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10528.878548] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10528.882063] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10528.885269] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10528.888723] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10528.891536] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10536.057024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10536.062380] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10536.115437] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10536.130538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10536.137546] RIP: 0010:native_safe_halt+0x2e/0x50 [10536.140879] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10536.165465] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10536.177529] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10536.179946] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10536.192326] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10536.198327] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10536.206571] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10536.211380] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10536.216573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10536.218324] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10536.230612] Call Trace: [10536.232579] [10536.233584] ? show_regs.cold.9+0x22/0x2f [10536.236445] ? watchdog_timer_fn.cold.6+0x62/0x10a [10536.244309] ? __hrtimer_run_queues+0x1c2/0x4d0 [10536.245784] ? watchdog+0x40/0x40 [10536.250335] ? hrtimer_interrupt+0x128/0x2c0 [10536.257516] ? smp_apic_timer_interrupt+0xa6/0x270 [10536.263017] ? apic_timer_interrupt+0xf/0x20 [10536.264241] [10536.264838] ? native_safe_halt+0x2e/0x50 [10536.269441] kvm_wait+0xcd/0xe0 [10536.270548] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10536.275024] do_raw_spin_lock+0x177/0x190 [10536.277814] _raw_spin_lock+0x21/0x30 [10536.278892] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10536.281246] ? do_raw_spin_unlock+0x75/0x190 [10536.283195] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10536.286054] ? do_wait_intr+0xf0/0xf0 [10536.287431] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10536.292481] mdc_close+0x53b/0xd70 [mdc] [10536.294298] lmv_close+0x18d/0x490 [lmv] [10536.295719] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10536.298607] ll_md_real_close+0xa5/0x2a0 [lustre] [10536.304986] ll_md_close+0x274/0x800 [lustre] [10536.308582] ll_file_release+0x128/0xc50 [lustre] [10536.312515] __fput+0x10d/0x440 [10536.314120] ____fput+0x12/0x20 [10536.315976] task_work_run+0xc6/0x110 [10536.317589] exit_to_usermode_loop+0x1dd/0x1f0 [10536.320769] do_syscall_64+0x3d6/0x440 [10536.323504] entry_SYSCALL_64_after_hwframe+0x49/0xae [10536.327608] RIP: 0033:0x7f1a63983cd8 [10536.332559] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10536.353642] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10536.358609] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10536.363667] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10536.371195] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10536.381703] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10536.387523] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10540.053881] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [10540.056532] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10540.106687] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10540.122774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10540.143613] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10540.152394] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10540.189324] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10540.195478] RAX: 000000000000710d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10540.213578] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10540.224951] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10540.233534] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10540.247378] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10540.259183] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10540.274148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10540.283056] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10540.289150] Call Trace: [10540.290083] [10540.290859] ? show_regs.cold.9+0x22/0x2f [10540.294615] ? watchdog_timer_fn.cold.6+0x62/0x10a [10540.300219] ? __hrtimer_run_queues+0x1c2/0x4d0 [10540.305554] ? watchdog+0x40/0x40 [10540.312304] ? hrtimer_interrupt+0x128/0x2c0 [10540.319842] ? smp_apic_timer_interrupt+0xa6/0x270 [10540.329093] ? apic_timer_interrupt+0xf/0x20 [10540.330659] [10540.331481] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10540.342621] do_raw_spin_lock+0x177/0x190 [10540.347367] _raw_spin_lock+0x21/0x30 [10540.351891] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10540.357054] ? do_raw_spin_unlock+0x75/0x190 [10540.358576] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10540.365246] ? do_wait_intr+0xf0/0xf0 [10540.367860] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10540.369737] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10540.378407] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10540.382324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10540.388704] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10540.390603] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10540.400474] mdc_intent_lock+0x353/0x7e0 [mdc] [10540.403081] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10540.410321] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10540.417446] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10540.422307] lmv_intent_open+0x338/0x1150 [lmv] [10540.428429] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10540.433586] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10540.441632] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10540.445604] ll_intent_lock+0xc5/0xaa0 [lustre] [10540.449349] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10540.457605] ll_lookup_it+0xe0e/0x2270 [lustre] [10540.459573] ? do_raw_spin_unlock+0x75/0x190 [10540.468633] ? _raw_spin_unlock+0x12/0x30 [10540.473031] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10540.483193] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10540.487907] ll_atomic_open+0x36a/0xd70 [lustre] [10540.491358] lookup_open+0x356/0xb40 [10540.492688] do_last+0x313/0xfc0 [10540.494984] ? nd_jump_root+0xe5/0x160 [10540.498057] ? path_init+0x437/0x520 [10540.499660] path_openat+0xf7/0x500 [10540.502968] ? do_raw_spin_unlock+0x75/0x190 [10540.507171] do_filp_open+0x99/0x140 [10540.508455] ? getname_flags+0x6e/0x330 [10540.520056] ? __check_object_size+0xff/0x256 [10540.523047] ? do_raw_spin_unlock+0x75/0x190 [10540.524561] ? _raw_spin_unlock+0x12/0x30 [10540.541038] do_sys_openat2+0x2b4/0x410 [10540.542421] do_sys_open+0x73/0xa0 [10540.553714] __x64_sys_openat+0x24/0x30 [10540.555077] do_syscall_64+0xc1/0x440 [10540.559456] entry_SYSCALL_64_after_hwframe+0x49/0xae [10540.566569] RIP: 0033:0x7f1a63983332 [10540.572383] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10540.595985] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10540.599880] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10540.603710] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10540.609810] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10540.626295] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10540.632660] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10542.641723] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [10542.644648] rcu: 3-...!: (5367568 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [10542.657075] (detected by 1, t=8520235 jiffies, g=269633, q=3066) [10542.666997] Sending NMI from CPU 1 to CPUs 3: [10542.677235] NMI backtrace for cpu 3 [10542.677243] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10542.677245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10542.677246] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10542.677248] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10542.677250] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [10542.677252] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10542.677254] RDX: 0000000000004001 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10542.677255] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10542.677256] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10542.677258] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10542.677259] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10542.677261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10542.677262] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10542.677263] Call Trace: [10542.677264] [10542.677265] ? show_regs.cold.9+0x22/0x2f [10542.677266] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10542.677267] ? nmi_cpu_backtrace_handler+0x15/0x40 [10542.677269] ? nmi_handle+0xbd/0x210 [10542.677270] ? default_do_nmi+0x4b/0x1b0 [10542.677271] ? do_nmi+0x280/0x380 [10542.677271] ? end_repeat_nmi+0x16/0x1f [10542.677272] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10542.677273] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10542.677275] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10542.677276] [10542.677277] do_raw_spin_lock+0x177/0x190 [10542.677278] _raw_spin_lock+0x21/0x30 [10542.677279] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10542.677281] obd_mod_free+0x46/0x140 [ptlrpc] [10542.677282] mdc_commit_open+0x91/0xb0 [mdc] [10542.677283] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10542.677284] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10542.677286] after_reply+0xe2c/0x1780 [ptlrpc] [10542.677287] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10542.677288] ? __next_timer_interrupt+0x160/0x160 [10542.677290] ptlrpcd+0xa82/0xe40 [ptlrpc] [10542.677291] ? do_wait_intr+0xf0/0xf0 [10542.677292] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10542.677293] kthread+0x1d1/0x200 [10542.677295] ? set_kthread_struct+0x70/0x70 [10542.677296] ret_from_fork+0x1f/0x30 [10542.678737] rcu: rcu_sched kthread starved for 8519881 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [10543.201780] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10543.234048] rcu: RCU grace-period kthread stack dump: [10543.237767] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10543.246459] Call Trace: [10543.248433] __schedule+0x351/0xcb0 [10543.253442] schedule+0xc0/0x180 [10543.257568] schedule_timeout+0xb4/0x190 [10543.265434] ? do_raw_spin_unlock+0x75/0x190 [10543.275995] ? __next_timer_interrupt+0x160/0x160 [10543.277728] ? prepare_to_swait_event+0xf3/0x140 [10543.287198] rcu_gp_kthread+0x8dd/0x15b0 [10543.288532] ? force_qs_rnp+0x2c0/0x2c0 [10543.294655] kthread+0x1d1/0x200 [10543.296351] ? set_kthread_struct+0x70/0x70 [10543.302991] ret_from_fork+0x1f/0x30 [10543.307388] rcu: Stack dump where RCU GP kthread last ran: [10543.315601] Sending NMI from CPU 1 to CPUs 2: [10543.324172] NMI backtrace for cpu 2 [10543.324178] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10543.324181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10543.324183] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10543.324185] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10543.324187] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 [10543.324189] RAX: 0000000000001fb2 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10543.324191] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10543.324192] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10543.324194] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10543.324195] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10543.324197] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10543.324198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10543.324200] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10543.324201] Call Trace: [10543.324202] [10543.324203] ? show_regs.cold.9+0x22/0x2f [10543.324204] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10543.324205] ? nmi_cpu_backtrace_handler+0x15/0x40 [10543.324206] ? nmi_handle+0xbd/0x210 [10543.324207] ? default_do_nmi+0x4b/0x1b0 [10543.324208] ? do_nmi+0x280/0x380 [10543.324209] ? end_repeat_nmi+0x16/0x1f [10543.324210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10543.324211] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10543.324212] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10543.324213] [10543.324214] do_raw_spin_lock+0x177/0x190 [10543.324214] _raw_spin_lock+0x21/0x30 [10543.324215] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10543.324216] ? do_raw_spin_unlock+0x75/0x190 [10543.324217] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10543.324218] ? do_wait_intr+0xf0/0xf0 [10543.324219] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10543.324219] mdc_close+0x53b/0xd70 [mdc] [10543.324220] lmv_close+0x18d/0x490 [lmv] [10543.324221] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10543.324223] ll_md_real_close+0xa5/0x2a0 [lustre] [10543.324223] ll_md_close+0x274/0x800 [lustre] [10543.324225] ll_file_release+0x128/0xc50 [lustre] [10543.324225] __fput+0x10d/0x440 [10543.324227] ____fput+0x12/0x20 [10543.324228] task_work_run+0xc6/0x110 [10543.324229] exit_to_usermode_loop+0x1dd/0x1f0 [10543.324230] do_syscall_64+0x3d6/0x440 [10543.324232] entry_SYSCALL_64_after_hwframe+0x49/0xae [10543.324233] RIP: 0033:0x7f1a63983cd8 [10543.324234] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10543.324237] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10543.324239] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10543.324241] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10543.324242] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10543.324244] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10543.324245] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10544.101689] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8521s! [10544.110191] Showing busy workqueues and worker pools: [10544.115195] workqueue events: flags=0x0 [10544.116702] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10544.128105] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10544.136311] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10544.143212] pending: sec_gc_main [ptlrpc] [10544.151106] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10544.155610] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10544.164117] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10544.168182] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10544.181829] workqueue events_long: flags=0x0 [10544.187268] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10544.195058] pending: lu_objects_flush [obdclass] [10544.198749] workqueue events_unbound: flags=0x2 [10544.201365] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10544.205474] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10544.214080] workqueue events_power_efficient: flags=0x80 [10544.217437] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10544.221305] pending: neigh_periodic_work, do_cache_clean [10544.224510] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10544.231462] pending: neigh_periodic_work, check_lifetime [10544.236300] workqueue mm_percpu_wq: flags=0x8 [10544.241043] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10544.249345] pending: vmstat_update [10544.257057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10544.264439] pending: vmstat_update [10544.266037] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10544.269510] pending: vmstat_update [10544.274028] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10544.278888] pending: vmstat_update [10544.281827] workqueue writeback: flags=0x4a [10544.284212] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10544.289922] pending: wb_workfn, wb_workfn [10544.294310] workqueue xprtiod: flags=0xa [10544.300117] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10544.302267] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10544.308308] workqueue ipv6_addrconf: flags=0x40008 [10544.310784] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10544.313947] pending: addrconf_verify_work [10544.315496] workqueue knbd0-recv: flags=0x1a [10544.317326] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10544.320139] in-flight: 647:recv_work [10544.321888] workqueue ptlrpc_pinger: flags=0x2 [10544.323637] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10544.326031] pending: ptlrpc_pinger_main [ptlrpc] [10544.328199] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10534s workers=2 idle: 652 [10568.057383] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [10568.058730] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [10568.058742] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10568.058750] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10568.059471] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10568.059477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10568.059479] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10568.059492] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10568.059497] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10568.059501] RAX: 0000000000007689 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10568.059504] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10568.059506] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10568.059508] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10568.059510] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10568.059513] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10568.059517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10568.059519] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10568.059528] Call Trace: [10568.059533] [10568.059539] ? show_regs.cold.9+0x22/0x2f [10568.059549] ? watchdog_timer_fn.cold.6+0x62/0x10a [10568.059555] ? __hrtimer_run_queues+0x1c2/0x4d0 [10568.059560] ? watchdog+0x40/0x40 [10568.059565] ? hrtimer_interrupt+0x128/0x2c0 [10568.059569] ? smp_apic_timer_interrupt+0xa6/0x270 [10568.059593] ? apic_timer_interrupt+0xf/0x20 [10568.059598] [10568.059601] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10568.059608] do_raw_spin_lock+0x177/0x190 [10568.059613] _raw_spin_lock+0x21/0x30 [10568.059618] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10568.060141] ? do_raw_spin_unlock+0x75/0x190 [10568.060193] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [10568.060147] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10568.060375] ? do_wait_intr+0xf0/0xf0 [10568.060384] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10568.060841] mdc_close+0x53b/0xd70 [mdc] [10568.060880] lmv_close+0x18d/0x490 [lmv] [10568.060977] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10568.061254] ll_md_real_close+0xa5/0x2a0 [lustre] [10568.062309] ll_md_close+0x274/0x800 [lustre] [10568.062399] ll_file_release+0x128/0xc50 [lustre] [10568.063525] __fput+0x10d/0x440 [10568.063538] ____fput+0x12/0x20 [10568.063543] task_work_run+0xc6/0x110 [10568.063550] exit_to_usermode_loop+0x1dd/0x1f0 [10568.063557] do_syscall_64+0x3d6/0x440 [10568.063562] entry_SYSCALL_64_after_hwframe+0x49/0xae [10568.063837] RIP: 0033:0x7f1a63983cd8 [10568.063846] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10568.063864] Modules linked in: lustre(O) [10568.063962] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10568.063968] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10568.063971] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10568.063974] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10568.063977] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10568.063980] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10568.067662] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10568.067706] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10568.067713] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10568.067715] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10568.067728] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10568.067732] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10568.067735] RAX: 0000000000006630 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10568.067738] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10568.067740] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10568.067742] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10568.067744] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10568.067747] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10568.067749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10568.067751] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10568.067759] Call Trace: [10568.067764] [10568.067768] ? show_regs.cold.9+0x22/0x2f [10568.067776] ? watchdog_timer_fn.cold.6+0x62/0x10a [10568.067782] ? __hrtimer_run_queues+0x1c2/0x4d0 [10568.067786] ? watchdog+0x40/0x40 [10568.067791] ? hrtimer_interrupt+0x128/0x2c0 [10568.067795] ? smp_apic_timer_interrupt+0xa6/0x270 [10568.067803] ? apic_timer_interrupt+0xf/0x20 [10568.067821] [10568.067824] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10568.067831] do_raw_spin_lock+0x177/0x190 [10568.067836] _raw_spin_lock+0x21/0x30 [10568.067841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10568.068103] ? do_raw_spin_unlock+0x75/0x190 [10568.068110] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10568.068507] ? do_wait_intr+0xf0/0xf0 [10568.068515] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10568.068720] mdc_close+0x53b/0xd70 [mdc] [10568.068761] lmv_close+0x18d/0x490 [lmv] [10568.068792] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10568.068961] ll_md_real_close+0xa5/0x2a0 [lustre] [10568.069043] ll_md_close+0x274/0x800 [lustre] [10568.069125] ll_file_release+0x128/0xc50 [lustre] [10568.069207] __fput+0x10d/0x440 [10568.069215] ____fput+0x12/0x20 [10568.069219] task_work_run+0xc6/0x110 [10568.069226] exit_to_usermode_loop+0x1dd/0x1f0 [10568.069233] do_syscall_64+0x3d6/0x440 [10568.069237] entry_SYSCALL_64_after_hwframe+0x49/0xae [10568.069244] RIP: 0033:0x7f1a63983cd8 [10568.069250] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10568.069254] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10568.069258] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10568.069260] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10568.069338] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10568.069341] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10568.069343] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10568.370867] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10568.391093] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10568.403280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10568.412145] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10568.418080] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10568.424167] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10568.429247] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10568.433062] RDX: 00000000000015ef RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10568.436154] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10568.438063] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10568.441129] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10568.444207] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10568.446559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10568.448917] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10568.451220] Call Trace: [10568.452263] [10568.453075] ? show_regs.cold.9+0x22/0x2f [10568.454053] ? watchdog_timer_fn.cold.6+0x62/0x10a [10568.455236] ? __hrtimer_run_queues+0x1c2/0x4d0 [10568.461676] ? watchdog+0x40/0x40 [10568.462566] ? hrtimer_interrupt+0x128/0x2c0 [10568.463920] ? smp_apic_timer_interrupt+0xa6/0x270 [10568.468897] ? apic_timer_interrupt+0xf/0x20 [10568.470401] [10568.472217] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10568.475768] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10568.477351] do_raw_spin_lock+0x177/0x190 [10568.478533] _raw_spin_lock+0x21/0x30 [10568.480246] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10568.481839] obd_mod_free+0x46/0x140 [ptlrpc] [10568.483102] mdc_commit_open+0x91/0xb0 [mdc] [10568.485722] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10568.487209] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10568.489038] after_reply+0xe2c/0x1780 [ptlrpc] [10568.490350] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10568.496271] ? __next_timer_interrupt+0x160/0x160 [10568.499183] ptlrpcd+0xa82/0xe40 [ptlrpc] [10568.501004] ? do_wait_intr+0xf0/0xf0 [10568.503047] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10568.504898] kthread+0x1d1/0x200 [10568.505834] ? set_kthread_struct+0x70/0x70 [10568.512735] ret_from_fork+0x1f/0x30 [10572.053594] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10572.064236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10572.118903] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10572.129373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10572.137144] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10572.147278] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10572.165110] RSP: 0018:ffffc38541c37500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10572.174337] RAX: 0000000000004b87 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10572.178741] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10572.189339] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10572.192909] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10572.195576] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10572.198270] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10572.201361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10572.203621] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10572.206109] Call Trace: [10572.207061] [10572.207858] ? show_regs.cold.9+0x22/0x2f [10572.209352] ? watchdog_timer_fn.cold.6+0x62/0x10a [10572.211209] ? __hrtimer_run_queues+0x1c2/0x4d0 [10572.212935] ? watchdog+0x40/0x40 [10572.215260] ? hrtimer_interrupt+0x128/0x2c0 [10572.222949] ? smp_apic_timer_interrupt+0xa6/0x270 [10572.228235] ? apic_timer_interrupt+0xf/0x20 [10572.233655] [10572.235824] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10572.238848] do_raw_spin_lock+0x177/0x190 [10572.242775] _raw_spin_lock+0x21/0x30 [10572.244186] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10572.249253] ? do_raw_spin_unlock+0x75/0x190 [10572.250835] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10572.263105] ? do_wait_intr+0xf0/0xf0 [10572.269361] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10572.274819] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10572.278143] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10572.282878] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10572.286960] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10572.291985] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10572.296635] mdc_intent_lock+0x353/0x7e0 [mdc] [10572.299102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10572.304246] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10572.314590] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10572.322662] lmv_intent_open+0x338/0x1150 [lmv] [10572.327668] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10572.333233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10572.337899] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10572.341058] ll_intent_lock+0xc5/0xaa0 [lustre] [10572.342586] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10572.344833] ll_lookup_it+0xe0e/0x2270 [lustre] [10572.347185] ? do_raw_spin_unlock+0x75/0x190 [10572.349301] ? _raw_spin_unlock+0x12/0x30 [10572.351794] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10572.354747] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10572.357666] ll_atomic_open+0x36a/0xd70 [lustre] [10572.360791] lookup_open+0x356/0xb40 [10572.362637] do_last+0x313/0xfc0 [10572.364202] ? nd_jump_root+0xe5/0x160 [10572.366333] ? path_init+0x437/0x520 [10572.368274] path_openat+0xf7/0x500 [10572.370118] ? do_raw_spin_unlock+0x75/0x190 [10572.372679] do_filp_open+0x99/0x140 [10572.374867] ? getname_flags+0x6e/0x330 [10572.376938] ? __check_object_size+0xff/0x256 [10572.378783] ? do_raw_spin_unlock+0x75/0x190 [10572.380807] ? _raw_spin_unlock+0x12/0x30 [10572.382566] do_sys_openat2+0x2b4/0x410 [10572.386798] do_sys_open+0x73/0xa0 [10572.388105] __x64_sys_openat+0x24/0x30 [10572.390786] do_syscall_64+0xc1/0x440 [10572.393324] entry_SYSCALL_64_after_hwframe+0x49/0xae [10572.397559] RIP: 0033:0x7f1a63983332 [10572.400254] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10572.413862] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10572.421251] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10572.424258] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10572.428524] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10572.431732] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10572.436207] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10574.817037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8551s! [10574.820394] Showing busy workqueues and worker pools: [10574.822127] workqueue events: flags=0x0 [10574.824918] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10574.829734] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10574.832869] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10574.835457] pending: sec_gc_main [ptlrpc] [10574.837292] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10574.840006] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10574.843305] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10574.846267] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10574.850124] workqueue events_long: flags=0x0 [10574.851900] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10574.854865] pending: lu_objects_flush [obdclass] [10574.857346] workqueue events_unbound: flags=0x2 [10574.858909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10574.861123] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10574.865193] workqueue events_power_efficient: flags=0x80 [10574.867014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10574.869518] pending: neigh_periodic_work, do_cache_clean [10574.871541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10574.873901] pending: neigh_periodic_work, check_lifetime [10574.875671] workqueue mm_percpu_wq: flags=0x8 [10574.876964] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10574.878918] pending: vmstat_update [10574.880296] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10574.882593] pending: vmstat_update [10574.884153] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10574.886255] pending: vmstat_update [10574.887986] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10574.890154] pending: vmstat_update [10574.891701] workqueue writeback: flags=0x4a [10574.893341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10574.895568] pending: wb_workfn, wb_workfn [10574.897663] workqueue xprtiod: flags=0xa [10574.898905] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10574.901165] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10574.903916] workqueue ipv6_addrconf: flags=0x40008 [10574.905385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10574.907319] pending: addrconf_verify_work [10574.908778] workqueue knbd0-recv: flags=0x1a [10574.910337] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10574.912705] in-flight: 647:recv_work [10574.914215] workqueue ptlrpc_pinger: flags=0x2 [10574.916622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10574.918506] pending: ptlrpc_pinger_main [ptlrpc] [10574.920521] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10564s workers=2 idle: 652 [10596.056447] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [10596.057469] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10596.058014] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [10596.058022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10596.058099] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10596.058104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10596.058108] RIP: 0010:native_safe_halt+0x2e/0x50 [10596.058122] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10596.058126] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10596.058132] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10596.058134] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10596.058137] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10596.058139] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10596.058142] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10596.058145] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10596.058149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10596.058161] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10596.058170] Call Trace: [10596.058175] [10596.058180] ? show_regs.cold.9+0x22/0x2f [10596.058190] ? watchdog_timer_fn.cold.6+0x62/0x10a [10596.058196] ? __hrtimer_run_queues+0x1c2/0x4d0 [10596.058200] ? watchdog+0x40/0x40 [10596.058205] ? hrtimer_interrupt+0x128/0x2c0 [10596.058209] ? smp_apic_timer_interrupt+0xa6/0x270 [10596.058216] ? apic_timer_interrupt+0xf/0x20 [10596.058221] [10596.058224] ? native_safe_halt+0x2e/0x50 [10596.058229] kvm_wait+0xcd/0xe0 [10596.058235] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10596.058243] do_raw_spin_lock+0x177/0x190 [10596.058248] _raw_spin_lock+0x21/0x30 [10596.059623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10596.059686] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10596.059692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10596.059695] RIP: 0010:native_safe_halt+0x2e/0x50 [10596.059706] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10596.059711] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10596.059714] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10596.059717] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10596.059719] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10596.059721] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10596.059724] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10596.059726] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10596.059729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10596.059731] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10596.059739] Call Trace: [10596.059743] [10596.059747] ? show_regs.cold.9+0x22/0x2f [10596.059755] ? watchdog_timer_fn.cold.6+0x62/0x10a [10596.059761] ? __hrtimer_run_queues+0x1c2/0x4d0 [10596.059764] ? watchdog+0x40/0x40 [10596.059768] ? hrtimer_interrupt+0x128/0x2c0 [10596.059771] ? smp_apic_timer_interrupt+0xa6/0x270 [10596.059777] ? apic_timer_interrupt+0xf/0x20 [10596.059781] [10596.059783] ? native_safe_halt+0x2e/0x50 [10596.059787] kvm_wait+0xcd/0xe0 [10596.059793] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10596.059800] do_raw_spin_lock+0x177/0x190 [10596.059804] _raw_spin_lock+0x21/0x30 [10596.059808] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10596.060069] ? do_raw_spin_unlock+0x75/0x190 [10596.060076] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10596.060695] ? do_wait_intr+0xf0/0xf0 [10596.060703] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10596.058253] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10596.061973] obd_mod_free+0x46/0x140 [ptlrpc] [10596.062185] mdc_commit_open+0x91/0xb0 [mdc] [10596.062225] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10596.062630] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10596.063151] after_reply+0xe2c/0x1780 [ptlrpc] [10596.063725] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10596.063921] ? __next_timer_interrupt+0x160/0x160 [10596.063932] ptlrpcd+0xa82/0xe40 [ptlrpc] [10596.064133] ? do_wait_intr+0xf0/0xf0 [10596.064143] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10596.066277] kthread+0x1d1/0x200 [10596.066286] ? set_kthread_struct+0x70/0x70 [10596.066290] ret_from_fork+0x1f/0x30 [10596.069139] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10596.069280] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10596.069287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10596.069291] RIP: 0010:native_safe_halt+0x2e/0x50 [10596.069303] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10596.069307] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10596.069312] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10596.069315] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [10596.069318] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10596.069320] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10596.069322] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10596.069326] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10596.069329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10596.069332] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10596.069388] Call Trace: [10596.069451] [10596.069458] ? show_regs.cold.9+0x22/0x2f [10596.069468] ? watchdog_timer_fn.cold.6+0x62/0x10a [10596.069474] ? __hrtimer_run_queues+0x1c2/0x4d0 [10596.069479] ? watchdog+0x40/0x40 [10596.069483] ? hrtimer_interrupt+0x128/0x2c0 [10596.069487] ? smp_apic_timer_interrupt+0xa6/0x270 [10596.069494] ? apic_timer_interrupt+0xf/0x20 [10596.069498] [10596.069501] ? native_safe_halt+0x2e/0x50 [10596.069513] kvm_wait+0xcd/0xe0 [10596.069521] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10596.069529] do_raw_spin_lock+0x177/0x190 [10596.069533] _raw_spin_lock+0x21/0x30 [10596.069537] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10596.069815] ? do_raw_spin_unlock+0x75/0x190 [10596.069822] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10596.070566] ? do_wait_intr+0xf0/0xf0 [10596.070574] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10596.079341] mdc_close+0x53b/0xd70 [mdc] [10596.079393] lmv_close+0x18d/0x490 [lmv] [10596.079425] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10596.079569] ll_md_real_close+0xa5/0x2a0 [lustre] [10596.079658] ll_md_close+0x274/0x800 [lustre] [10596.079746] ll_file_release+0x128/0xc50 [lustre] [10596.079831] __fput+0x10d/0x440 [10596.079842] ____fput+0x12/0x20 [10596.079847] task_work_run+0xc6/0x110 [10596.079855] exit_to_usermode_loop+0x1dd/0x1f0 [10596.079862] do_syscall_64+0x3d6/0x440 [10596.079866] entry_SYSCALL_64_after_hwframe+0x49/0xae [10596.079875] RIP: 0033:0x7f1a63983cd8 [10596.079882] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10596.079886] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10596.079891] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10596.079894] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10596.079897] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10596.079899] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10596.079902] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10596.754686] mdc_close+0x53b/0xd70 [mdc] [10596.759404] lmv_close+0x18d/0x490 [lmv] [10596.762296] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10596.766241] ll_md_real_close+0xa5/0x2a0 [lustre] [10596.768806] ll_md_close+0x274/0x800 [lustre] [10596.774425] ll_file_release+0x128/0xc50 [lustre] [10596.776007] __fput+0x10d/0x440 [10596.777345] ____fput+0x12/0x20 [10596.779376] task_work_run+0xc6/0x110 [10596.781487] exit_to_usermode_loop+0x1dd/0x1f0 [10596.786462] do_syscall_64+0x3d6/0x440 [10596.788726] entry_SYSCALL_64_after_hwframe+0x49/0xae [10596.798482] RIP: 0033:0x7f1a63983cd8 [10596.805940] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10596.830899] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10596.837893] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10596.849744] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10596.865032] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10596.869647] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10596.872655] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10600.054344] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [10600.061894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10600.100095] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10600.105766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10600.111530] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10600.114940] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10600.130879] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10600.139007] RAX: 0000000000001793 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10600.143286] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10600.150735] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10600.155047] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10600.159210] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10600.163503] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10600.167645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10600.170678] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10600.174296] Call Trace: [10600.175539] [10600.178385] ? show_regs.cold.9+0x22/0x2f [10600.182585] ? watchdog_timer_fn.cold.6+0x62/0x10a [10600.193395] ? __hrtimer_run_queues+0x1c2/0x4d0 [10600.197403] ? watchdog+0x40/0x40 [10600.199450] ? hrtimer_interrupt+0x128/0x2c0 [10600.203959] ? smp_apic_timer_interrupt+0xa6/0x270 [10600.206719] ? apic_timer_interrupt+0xf/0x20 [10600.209369] [10600.210708] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10600.214227] do_raw_spin_lock+0x177/0x190 [10600.218732] _raw_spin_lock+0x21/0x30 [10600.220023] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10600.228056] ? do_raw_spin_unlock+0x75/0x190 [10600.233721] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10600.235836] ? do_wait_intr+0xf0/0xf0 [10600.236932] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10600.246364] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10600.249448] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10600.258768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10600.263396] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10600.266363] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10600.275235] mdc_intent_lock+0x353/0x7e0 [mdc] [10600.281471] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10600.291391] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10600.307738] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10600.309714] lmv_intent_open+0x338/0x1150 [lmv] [10600.314916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10600.319285] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10600.322904] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10600.325207] ll_intent_lock+0xc5/0xaa0 [lustre] [10600.328087] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10600.331042] ll_lookup_it+0xe0e/0x2270 [lustre] [10600.333398] ? do_raw_spin_unlock+0x75/0x190 [10600.336991] ? _raw_spin_unlock+0x12/0x30 [10600.341843] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10600.346768] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10600.352721] ll_atomic_open+0x36a/0xd70 [lustre] [10600.356559] lookup_open+0x356/0xb40 [10600.358216] do_last+0x313/0xfc0 [10600.360438] ? nd_jump_root+0xe5/0x160 [10600.362958] ? path_init+0x437/0x520 [10600.366441] path_openat+0xf7/0x500 [10600.369471] ? do_raw_spin_unlock+0x75/0x190 [10600.372362] do_filp_open+0x99/0x140 [10600.373302] ? getname_flags+0x6e/0x330 [10600.380642] ? __check_object_size+0xff/0x256 [10600.381928] ? do_raw_spin_unlock+0x75/0x190 [10600.390156] ? _raw_spin_unlock+0x12/0x30 [10600.392084] do_sys_openat2+0x2b4/0x410 [10600.394409] do_sys_open+0x73/0xa0 [10600.396296] __x64_sys_openat+0x24/0x30 [10600.399177] do_syscall_64+0xc1/0x440 [10600.401211] entry_SYSCALL_64_after_hwframe+0x49/0xae [10600.404799] RIP: 0033:0x7f1a63983332 [10600.407704] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10600.420940] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10600.425901] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10600.431802] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10600.437958] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10600.448092] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10600.452408] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10605.025114] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8582s! [10605.029185] Showing busy workqueues and worker pools: [10605.031415] workqueue events: flags=0x0 [10605.033367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10605.037238] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10605.040807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10605.045618] pending: sec_gc_main [ptlrpc] [10605.048467] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10605.052198] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10605.057451] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10605.060869] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10605.065748] workqueue events_long: flags=0x0 [10605.068493] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10605.072235] pending: lu_objects_flush [obdclass] [10605.073918] workqueue events_unbound: flags=0x2 [10605.075356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10605.077266] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10605.081011] workqueue events_power_efficient: flags=0x80 [10605.082801] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10605.084981] pending: neigh_periodic_work, do_cache_clean [10605.086934] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10605.089482] pending: neigh_periodic_work, check_lifetime [10605.091884] workqueue mm_percpu_wq: flags=0x8 [10605.093634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10605.095951] pending: vmstat_update [10605.097276] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10605.099719] pending: vmstat_update [10605.101319] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10605.103500] pending: vmstat_update [10605.104684] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10605.106811] pending: vmstat_update [10605.108128] workqueue writeback: flags=0x4a [10605.109466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10605.111425] pending: wb_workfn, wb_workfn [10605.113040] workqueue xprtiod: flags=0xa [10605.114498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10605.116788] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10605.119612] workqueue ipv6_addrconf: flags=0x40008 [10605.121273] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10605.123260] pending: addrconf_verify_work [10605.124742] workqueue knbd0-recv: flags=0x1a [10605.126338] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10605.128673] in-flight: 647:recv_work [10605.130178] workqueue ptlrpc_pinger: flags=0x2 [10605.131727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10605.133774] pending: ptlrpc_pinger_main [ptlrpc] [10605.135941] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10595s workers=2 idle: 652 [10624.058700] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10624.061949] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10624.065905] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10624.065918] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10624.065994] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10624.066000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10624.066002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10624.066016] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10624.066020] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10624.066024] RAX: 0000000000007c84 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10624.066027] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10624.066029] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10624.066031] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10624.066033] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10624.066036] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10624.066039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10624.066041] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10624.066049] Call Trace: [10624.066054] [10624.066059] ? show_regs.cold.9+0x22/0x2f [10624.066067] ? watchdog_timer_fn.cold.6+0x62/0x10a [10624.066073] ? __hrtimer_run_queues+0x1c2/0x4d0 [10624.066076] ? watchdog+0x40/0x40 [10624.066090] ? hrtimer_interrupt+0x128/0x2c0 [10624.066094] ? smp_apic_timer_interrupt+0xa6/0x270 [10624.066116] ? apic_timer_interrupt+0xf/0x20 [10624.066122] [10624.066125] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10624.066132] do_raw_spin_lock+0x177/0x190 [10624.066136] _raw_spin_lock+0x21/0x30 [10624.066333] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10624.066632] ? do_raw_spin_unlock+0x75/0x190 [10624.066639] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10624.066859] ? do_wait_intr+0xf0/0xf0 [10624.066867] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10624.067095] mdc_close+0x53b/0xd70 [mdc] [10624.067262] lmv_close+0x18d/0x490 [lmv] [10624.067299] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10624.067408] ll_md_real_close+0xa5/0x2a0 [lustre] [10624.067540] ll_md_close+0x274/0x800 [lustre] [10624.067635] ll_file_release+0x128/0xc50 [lustre] [10624.067728] __fput+0x10d/0x440 [10624.067801] ____fput+0x12/0x20 [10624.067808] task_work_run+0xc6/0x110 [10624.067815] exit_to_usermode_loop+0x1dd/0x1f0 [10624.067821] do_syscall_64+0x3d6/0x440 [10624.067825] entry_SYSCALL_64_after_hwframe+0x49/0xae [10624.067831] RIP: 0033:0x7f1a63983cd8 [10624.067838] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10624.067842] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10624.067847] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10624.067849] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10624.067851] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10624.067854] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10624.067856] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10624.073411] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10624.073498] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10624.073506] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10624.073509] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10624.073523] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10624.073527] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10624.073532] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10624.073535] RDX: 000000000000670c RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10624.073537] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10624.073540] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10624.073543] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10624.073546] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10624.073549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10624.073552] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10624.073561] Call Trace: [10624.073566] [10624.073571] ? show_regs.cold.9+0x22/0x2f [10624.073580] ? watchdog_timer_fn.cold.6+0x62/0x10a [10624.073586] ? __hrtimer_run_queues+0x1c2/0x4d0 [10624.073591] ? watchdog+0x40/0x40 [10624.073595] ? hrtimer_interrupt+0x128/0x2c0 [10624.073599] ? smp_apic_timer_interrupt+0xa6/0x270 [10624.073607] ? apic_timer_interrupt+0xf/0x20 [10624.073612] [10624.073615] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10624.073620] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10624.073625] do_raw_spin_lock+0x177/0x190 [10624.073630] _raw_spin_lock+0x21/0x30 [10624.073635] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10624.095486] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10624.095561] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10624.095568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10624.095571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10624.095584] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10624.095588] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10624.095593] RAX: 0000000000005608 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10624.095595] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10624.095597] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10624.095600] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10624.095603] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10624.095606] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10624.095610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10624.095612] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10624.095621] Call Trace: [10624.095625] [10624.095631] ? show_regs.cold.9+0x22/0x2f [10624.095640] ? watchdog_timer_fn.cold.6+0x62/0x10a [10624.095646] ? __hrtimer_run_queues+0x1c2/0x4d0 [10624.095650] ? watchdog+0x40/0x40 [10624.095654] ? hrtimer_interrupt+0x128/0x2c0 [10624.095658] ? smp_apic_timer_interrupt+0xa6/0x270 [10624.095665] ? apic_timer_interrupt+0xf/0x20 [10624.095670] [10624.095673] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10624.095678] do_raw_spin_lock+0x177/0x190 [10624.095683] _raw_spin_lock+0x21/0x30 [10624.095687] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10624.096115] ? do_raw_spin_unlock+0x75/0x190 [10624.107842] obd_mod_free+0x46/0x140 [ptlrpc] [10624.108349] mdc_commit_open+0x91/0xb0 [mdc] [10624.108385] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10624.108593] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10624.108799] after_reply+0xe2c/0x1780 [ptlrpc] [10624.109010] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10624.109201] ? __next_timer_interrupt+0x160/0x160 [10624.109212] ptlrpcd+0xa82/0xe40 [ptlrpc] [10624.109413] ? do_wait_intr+0xf0/0xf0 [10624.109422] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10624.109619] kthread+0x1d1/0x200 [10624.109625] ? set_kthread_struct+0x70/0x70 [10624.109629] ret_from_fork+0x1f/0x30 [10625.104430] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10625.110679] ? do_wait_intr+0xf0/0xf0 [10625.115723] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10625.120754] mdc_close+0x53b/0xd70 [mdc] [10625.122751] lmv_close+0x18d/0x490 [lmv] [10625.127529] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10625.132458] ll_md_real_close+0xa5/0x2a0 [lustre] [10625.136155] ll_md_close+0x274/0x800 [lustre] [10625.139748] ll_file_release+0x128/0xc50 [lustre] [10625.151729] __fput+0x10d/0x440 [10625.156645] ____fput+0x12/0x20 [10625.159536] task_work_run+0xc6/0x110 [10625.162568] exit_to_usermode_loop+0x1dd/0x1f0 [10625.166419] do_syscall_64+0x3d6/0x440 [10625.169988] entry_SYSCALL_64_after_hwframe+0x49/0xae [10625.179606] RIP: 0033:0x7f1a63983cd8 [10625.185568] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10625.205060] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10625.218786] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10625.231298] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10625.242850] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10625.256908] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10625.267757] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10632.053265] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10632.057254] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10632.079486] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10632.085000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10632.088928] RIP: 0010:native_safe_halt+0x2e/0x50 [10632.091186] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10632.098488] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10632.101531] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10632.104330] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [10632.107211] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10632.110010] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10632.112369] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10632.114921] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10632.117416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10632.119416] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10632.122009] Call Trace: [10632.122873] [10632.123409] ? show_regs.cold.9+0x22/0x2f [10632.124980] ? watchdog_timer_fn.cold.6+0x62/0x10a [10632.126459] ? __hrtimer_run_queues+0x1c2/0x4d0 [10632.128186] ? watchdog+0x40/0x40 [10632.129292] ? hrtimer_interrupt+0x128/0x2c0 [10632.130988] ? smp_apic_timer_interrupt+0xa6/0x270 [10632.132444] ? apic_timer_interrupt+0xf/0x20 [10632.134070] [10632.135115] ? native_safe_halt+0x2e/0x50 [10632.136448] kvm_wait+0xcd/0xe0 [10632.145774] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10632.152283] do_raw_spin_lock+0x177/0x190 [10632.153527] _raw_spin_lock+0x21/0x30 [10632.155047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10632.157129] ? do_raw_spin_unlock+0x75/0x190 [10632.158930] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10632.160572] ? do_wait_intr+0xf0/0xf0 [10632.161931] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10632.163611] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10632.165508] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10632.167570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10632.169402] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10632.171417] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10632.173038] mdc_intent_lock+0x353/0x7e0 [mdc] [10632.174607] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10632.176536] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10632.178885] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10632.180468] lmv_intent_open+0x338/0x1150 [lmv] [10632.182031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10632.183646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10632.185455] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10632.187015] ll_intent_lock+0xc5/0xaa0 [lustre] [10632.188446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10632.190619] ll_lookup_it+0xe0e/0x2270 [lustre] [10632.192331] ? do_raw_spin_unlock+0x75/0x190 [10632.200638] ? _raw_spin_unlock+0x12/0x30 [10632.202464] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10632.206481] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10632.212166] ll_atomic_open+0x36a/0xd70 [lustre] [10632.214661] lookup_open+0x356/0xb40 [10632.218391] do_last+0x313/0xfc0 [10632.220097] ? nd_jump_root+0xe5/0x160 [10632.221269] ? path_init+0x437/0x520 [10632.222474] path_openat+0xf7/0x500 [10632.223616] ? do_raw_spin_unlock+0x75/0x190 [10632.225345] do_filp_open+0x99/0x140 [10632.226498] ? getname_flags+0x6e/0x330 [10632.228054] ? __check_object_size+0xff/0x256 [10632.229432] ? do_raw_spin_unlock+0x75/0x190 [10632.231047] ? _raw_spin_unlock+0x12/0x30 [10632.232439] do_sys_openat2+0x2b4/0x410 [10632.234164] do_sys_open+0x73/0xa0 [10632.238227] __x64_sys_openat+0x24/0x30 [10632.241236] do_syscall_64+0xc1/0x440 [10632.243457] entry_SYSCALL_64_after_hwframe+0x49/0xae [10632.247321] RIP: 0033:0x7f1a63983332 [10632.249420] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10632.263591] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10632.276290] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10632.286430] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10632.295425] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10632.310048] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10632.315352] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10635.238091] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8612s! [10635.255575] Showing busy workqueues and worker pools: [10635.270531] workqueue events: flags=0x0 [10635.274205] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10635.290158] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10635.310853] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10635.323680] pending: sec_gc_main [ptlrpc] [10635.333865] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10635.339229] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10635.363565] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10635.373045] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10635.379205] workqueue events_long: flags=0x0 [10635.382249] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10635.394116] pending: lu_objects_flush [obdclass] [10635.399064] workqueue events_unbound: flags=0x2 [10635.402390] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10635.411138] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10635.425275] workqueue events_power_efficient: flags=0x80 [10635.432144] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10635.443961] pending: neigh_periodic_work, do_cache_clean [10635.453328] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10635.462686] pending: neigh_periodic_work, check_lifetime [10635.476497] workqueue mm_percpu_wq: flags=0x8 [10635.485272] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10635.499283] pending: vmstat_update [10635.504543] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10635.516044] pending: vmstat_update [10635.524823] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10635.537050] pending: vmstat_update [10635.541625] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10635.555449] pending: vmstat_update [10635.558251] workqueue writeback: flags=0x4a [10635.563468] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10635.573410] pending: wb_workfn, wb_workfn [10635.575122] workqueue xprtiod: flags=0xa [10635.578528] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10635.581592] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10635.592169] workqueue ipv6_addrconf: flags=0x40008 [10635.600741] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10635.611827] pending: addrconf_verify_work [10635.612818] workqueue knbd0-recv: flags=0x1a [10635.626333] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10635.634949] in-flight: 647:recv_work [10635.636321] workqueue ptlrpc_pinger: flags=0x2 [10635.643597] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10635.658573] pending: ptlrpc_pinger_main [ptlrpc] [10635.665053] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10625s workers=2 idle: 652 [10652.056027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10652.059227] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10652.059371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10652.059540] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10652.059547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10652.059549] RIP: 0010:native_safe_halt+0x2e/0x50 [10652.059569] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10652.059573] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10652.059578] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10652.059761] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10652.059765] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10652.059768] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10652.059770] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10652.059775] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10652.059778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10652.059781] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10652.059790] Call Trace: [10652.059808] [10652.059815] ? show_regs.cold.9+0x22/0x2f [10652.059825] ? watchdog_timer_fn.cold.6+0x62/0x10a [10652.059832] ? __hrtimer_run_queues+0x1c2/0x4d0 [10652.059837] ? watchdog+0x40/0x40 [10652.059840] ? hrtimer_interrupt+0x128/0x2c0 [10652.059844] ? smp_apic_timer_interrupt+0xa6/0x270 [10652.059851] ? apic_timer_interrupt+0xf/0x20 [10652.059855] [10652.059858] ? native_safe_halt+0x2e/0x50 [10652.059861] kvm_wait+0xcd/0xe0 [10652.059868] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10652.059876] do_raw_spin_lock+0x177/0x190 [10652.059880] _raw_spin_lock+0x21/0x30 [10652.059883] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10652.060228] ? do_raw_spin_unlock+0x75/0x190 [10652.060235] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10652.060433] ? do_wait_intr+0xf0/0xf0 [10652.060441] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10652.062200] mdc_close+0x53b/0xd70 [mdc] [10652.062244] lmv_close+0x18d/0x490 [lmv] [10652.062276] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10652.062393] ll_md_real_close+0xa5/0x2a0 [lustre] [10652.062475] ll_md_close+0x274/0x800 [lustre] [10652.062557] ll_file_release+0x128/0xc50 [lustre] [10652.062855] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10652.062862] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10652.062932] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10652.062938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10652.062941] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10652.062953] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10652.062957] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10652.062961] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10652.062964] RDX: 0000000000004e0d RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10652.062971] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10652.062973] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10652.062976] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10652.062980] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10652.062983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10652.062985] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10652.062992] Call Trace: [10652.062997] [10652.062999] ? show_regs.cold.9+0x22/0x2f [10652.063009] ? watchdog_timer_fn.cold.6+0x62/0x10a [10652.063015] ? __hrtimer_run_queues+0x1c2/0x4d0 [10652.063019] ? watchdog+0x40/0x40 [10652.063023] ? hrtimer_interrupt+0x128/0x2c0 [10652.063026] ? smp_apic_timer_interrupt+0xa6/0x270 [10652.063036] ? apic_timer_interrupt+0xf/0x20 [10652.063047] [10652.063049] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10652.063054] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10652.063058] do_raw_spin_lock+0x177/0x190 [10652.063063] _raw_spin_lock+0x21/0x30 [10652.063069] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10652.064087] obd_mod_free+0x46/0x140 [ptlrpc] [10652.065076] mdc_commit_open+0x91/0xb0 [mdc] [10652.065136] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10652.065376] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10652.065824] after_reply+0xe2c/0x1780 [ptlrpc] [10652.066078] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10652.066436] ? __next_timer_interrupt+0x160/0x160 [10652.066450] ptlrpcd+0xa82/0xe40 [ptlrpc] [10652.067256] ? do_wait_intr+0xf0/0xf0 [10652.067267] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10652.068435] Modules linked in: lustre(O) [10652.068456] kthread+0x1d1/0x200 [10652.068464] ? set_kthread_struct+0x70/0x70 [10652.068468] ret_from_fork+0x1f/0x30 [10652.098458] __fput+0x10d/0x440 [10652.098481] ____fput+0x12/0x20 [10652.098486] task_work_run+0xc6/0x110 [10652.098493] exit_to_usermode_loop+0x1dd/0x1f0 [10652.098500] do_syscall_64+0x3d6/0x440 [10652.098505] entry_SYSCALL_64_after_hwframe+0x49/0xae [10652.098513] RIP: 0033:0x7f1a63983cd8 [10652.098565] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10652.098571] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10652.098576] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10652.098579] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10652.098581] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10652.098583] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10652.098584] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10652.576901] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10652.607968] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10652.618498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10652.629903] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10652.634161] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10652.656520] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10652.664922] RAX: 0000000000003975 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10652.668478] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10652.674363] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10652.690634] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10652.711912] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10652.722574] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10652.736276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10652.741700] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10652.754138] Call Trace: [10652.759165] [10652.761556] ? show_regs.cold.9+0x22/0x2f [10652.768712] ? watchdog_timer_fn.cold.6+0x62/0x10a [10652.772009] ? __hrtimer_run_queues+0x1c2/0x4d0 [10652.783380] ? watchdog+0x40/0x40 [10652.791496] ? hrtimer_interrupt+0x128/0x2c0 [10652.794092] ? smp_apic_timer_interrupt+0xa6/0x270 [10652.800984] ? apic_timer_interrupt+0xf/0x20 [10652.810332] [10652.814526] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10652.818772] do_raw_spin_lock+0x177/0x190 [10652.822122] _raw_spin_lock+0x21/0x30 [10652.824185] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10652.834913] ? do_raw_spin_unlock+0x75/0x190 [10652.837498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10652.852401] ? do_wait_intr+0xf0/0xf0 [10652.858513] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10652.865896] mdc_close+0x53b/0xd70 [mdc] [10652.871049] lmv_close+0x18d/0x490 [lmv] [10652.875008] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10652.886077] ll_md_real_close+0xa5/0x2a0 [lustre] [10652.892105] ll_md_close+0x274/0x800 [lustre] [10652.900091] ll_file_release+0x128/0xc50 [lustre] [10652.911294] __fput+0x10d/0x440 [10652.913426] ____fput+0x12/0x20 [10652.917992] task_work_run+0xc6/0x110 [10652.927293] exit_to_usermode_loop+0x1dd/0x1f0 [10652.930886] do_syscall_64+0x3d6/0x440 [10652.936347] entry_SYSCALL_64_after_hwframe+0x49/0xae [10652.946046] RIP: 0033:0x7f1a63983cd8 [10652.953524] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10652.976747] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10652.982403] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10652.987795] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10653.001036] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10653.014483] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10653.028015] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10660.053028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10660.055116] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10660.082165] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10660.100641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10660.113117] RIP: 0010:native_safe_halt+0x2e/0x50 [10660.119497] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10660.134199] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10660.137354] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10660.140153] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [10660.142799] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10660.145340] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10660.148310] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10660.151090] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10660.154685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10660.157136] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10660.160222] Call Trace: [10660.161270] [10660.162195] ? show_regs.cold.9+0x22/0x2f [10660.164006] ? watchdog_timer_fn.cold.6+0x62/0x10a [10660.166154] ? __hrtimer_run_queues+0x1c2/0x4d0 [10660.167849] ? watchdog+0x40/0x40 [10660.169062] ? hrtimer_interrupt+0x128/0x2c0 [10660.170759] ? smp_apic_timer_interrupt+0xa6/0x270 [10660.172433] ? apic_timer_interrupt+0xf/0x20 [10660.174505] [10660.175254] ? native_safe_halt+0x2e/0x50 [10660.176893] kvm_wait+0xcd/0xe0 [10660.178170] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10660.180311] do_raw_spin_lock+0x177/0x190 [10660.182245] _raw_spin_lock+0x21/0x30 [10660.183710] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10660.185696] ? do_raw_spin_unlock+0x75/0x190 [10660.187257] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10660.190043] ? do_wait_intr+0xf0/0xf0 [10660.191679] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10660.194160] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10660.196245] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10660.201083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10660.203318] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10660.205337] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10660.207160] mdc_intent_lock+0x353/0x7e0 [mdc] [10660.208963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10660.212813] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10660.219195] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10660.223976] lmv_intent_open+0x338/0x1150 [lmv] [10660.225727] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10660.231281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10660.237358] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10660.241365] ll_intent_lock+0xc5/0xaa0 [lustre] [10660.244641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10660.248024] ll_lookup_it+0xe0e/0x2270 [lustre] [10660.252910] ? do_raw_spin_unlock+0x75/0x190 [10660.255101] ? _raw_spin_unlock+0x12/0x30 [10660.258230] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10660.266605] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10660.268360] ll_atomic_open+0x36a/0xd70 [lustre] [10660.271861] lookup_open+0x356/0xb40 [10660.273710] do_last+0x313/0xfc0 [10660.277075] ? nd_jump_root+0xe5/0x160 [10660.281027] ? path_init+0x437/0x520 [10660.283989] path_openat+0xf7/0x500 [10660.286637] ? do_raw_spin_unlock+0x75/0x190 [10660.288713] do_filp_open+0x99/0x140 [10660.291793] ? getname_flags+0x6e/0x330 [10660.295446] ? __check_object_size+0xff/0x256 [10660.296922] ? do_raw_spin_unlock+0x75/0x190 [10660.308435] ? _raw_spin_unlock+0x12/0x30 [10660.309982] do_sys_openat2+0x2b4/0x410 [10660.315613] do_sys_open+0x73/0xa0 [10660.317263] __x64_sys_openat+0x24/0x30 [10660.321242] do_syscall_64+0xc1/0x440 [10660.323222] entry_SYSCALL_64_after_hwframe+0x49/0xae [10660.337279] RIP: 0033:0x7f1a63983332 [10660.341142] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10660.362744] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10660.370859] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10660.382746] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10660.388866] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10660.391111] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10660.393539] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10665.953048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8642s! [10665.956054] Showing busy workqueues and worker pools: [10665.959029] workqueue events: flags=0x0 [10665.960406] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10665.966919] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10665.970988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10665.975619] pending: sec_gc_main [ptlrpc] [10665.979956] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10665.985072] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10665.988610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10665.991064] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10665.997188] workqueue events_long: flags=0x0 [10665.998620] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10666.001070] pending: lu_objects_flush [obdclass] [10666.004027] workqueue events_unbound: flags=0x2 [10666.006153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10666.009171] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10666.014689] workqueue events_power_efficient: flags=0x80 [10666.017403] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10666.020644] pending: neigh_periodic_work, do_cache_clean [10666.023156] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10666.026566] pending: neigh_periodic_work, check_lifetime [10666.029150] workqueue mm_percpu_wq: flags=0x8 [10666.032274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10666.034930] pending: vmstat_update [10666.036602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10666.039127] pending: vmstat_update [10666.042114] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10666.044256] pending: vmstat_update [10666.046825] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10666.049247] pending: vmstat_update [10666.051268] workqueue writeback: flags=0x4a [10666.053252] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10666.056647] pending: wb_workfn, wb_workfn [10666.060036] workqueue xprtiod: flags=0xa [10666.061572] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10666.065790] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10666.069899] workqueue ipv6_addrconf: flags=0x40008 [10666.072437] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10666.075971] pending: addrconf_verify_work [10666.081015] workqueue knbd0-recv: flags=0x1a [10666.083066] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10666.087621] in-flight: 647:recv_work [10666.090676] workqueue ptlrpc_pinger: flags=0x2 [10666.093144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10666.095868] pending: ptlrpc_pinger_main [ptlrpc] [10666.098030] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10656s workers=2 idle: 652 [10680.060715] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10680.061425] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10680.062687] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10680.062697] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10680.062764] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10680.062771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10680.062774] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10680.062788] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10680.062792] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10680.062797] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10680.063122] RDX: 00000000000059aa RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10680.063127] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10680.063130] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10680.063133] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10680.063136] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10680.063139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10680.063141] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10680.063149] Call Trace: [10680.063154] [10680.063161] ? show_regs.cold.9+0x22/0x2f [10680.063171] ? watchdog_timer_fn.cold.6+0x62/0x10a [10680.063178] ? __hrtimer_run_queues+0x1c2/0x4d0 [10680.063182] ? watchdog+0x40/0x40 [10680.063187] ? hrtimer_interrupt+0x128/0x2c0 [10680.063191] ? smp_apic_timer_interrupt+0xa6/0x270 [10680.065562] ? apic_timer_interrupt+0xf/0x20 [10680.065587] [10680.065591] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10680.065598] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10680.065603] do_raw_spin_lock+0x177/0x190 [10680.065609] _raw_spin_lock+0x21/0x30 [10680.065615] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10680.067062] obd_mod_free+0x46/0x140 [ptlrpc] [10680.068217] mdc_commit_open+0x91/0xb0 [mdc] [10680.068255] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10680.068467] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10680.068662] after_reply+0xe2c/0x1780 [ptlrpc] [10680.068857] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10680.071437] ? __next_timer_interrupt+0x160/0x160 [10680.071672] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10680.071743] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10680.071751] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10680.071754] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10680.071766] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10680.071770] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10680.071774] RAX: 0000000000003c8e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10680.071777] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10680.072853] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10680.072856] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10680.072858] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10680.072862] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10680.072866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10680.072868] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10680.072877] Call Trace: [10680.072883] [10680.072887] ? show_regs.cold.9+0x22/0x2f [10680.072897] ? watchdog_timer_fn.cold.6+0x62/0x10a [10680.072904] ? __hrtimer_run_queues+0x1c2/0x4d0 [10680.072908] ? watchdog+0x40/0x40 [10680.072912] ? hrtimer_interrupt+0x128/0x2c0 [10680.072917] ? smp_apic_timer_interrupt+0xa6/0x270 [10680.072924] ? apic_timer_interrupt+0xf/0x20 [10680.072928] [10680.072931] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10680.072936] do_raw_spin_lock+0x177/0x190 [10680.072941] _raw_spin_lock+0x21/0x30 [10680.072946] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10680.073967] ? do_raw_spin_unlock+0x75/0x190 [10680.073974] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10680.074182] ? do_wait_intr+0xf0/0xf0 [10680.074189] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10680.074399] mdc_close+0x53b/0xd70 [mdc] [10680.071448] ptlrpcd+0xa82/0xe40 [ptlrpc] [10680.074602] ? do_wait_intr+0xf0/0xf0 [10680.074610] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10680.074737] lmv_close+0x18d/0x490 [lmv] [10680.075722] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10680.075917] ll_md_real_close+0xa5/0x2a0 [lustre] [10680.076169] ll_md_close+0x274/0x800 [lustre] [10680.076252] ll_file_release+0x128/0xc50 [lustre] [10680.076355] __fput+0x10d/0x440 [10680.076364] ____fput+0x12/0x20 [10680.076369] task_work_run+0xc6/0x110 [10680.076378] exit_to_usermode_loop+0x1dd/0x1f0 [10680.076385] do_syscall_64+0x3d6/0x440 [10680.076390] entry_SYSCALL_64_after_hwframe+0x49/0xae [10680.076399] RIP: 0033:0x7f1a63983cd8 [10680.076406] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10680.076411] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10680.076415] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10680.076417] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10680.076419] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10680.076421] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10680.076423] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10680.083886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac [10680.090548] kthread+0x1d1/0x200 [10680.090569] ? set_kthread_struct+0x70/0x70 [10680.090574] ret_from_fork+0x1f/0x30 [10680.908980] rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10680.924259] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10680.930968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10680.942111] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10680.947888] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10680.960290] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10680.964786] RAX: 0000000000002785 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10680.967855] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10680.971554] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10680.975282] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10680.978468] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10680.981823] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10680.994057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10680.998897] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10681.002666] Call Trace: [10681.004028] [10681.005105] ? show_regs.cold.9+0x22/0x2f [10681.007674] ? watchdog_timer_fn.cold.6+0x62/0x10a [10681.010132] ? __hrtimer_run_queues+0x1c2/0x4d0 [10681.012586] ? watchdog+0x40/0x40 [10681.014523] ? hrtimer_interrupt+0x128/0x2c0 [10681.021319] ? smp_apic_timer_interrupt+0xa6/0x270 [10681.024966] ? apic_timer_interrupt+0xf/0x20 [10681.026724] [10681.028842] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10681.031446] do_raw_spin_lock+0x177/0x190 [10681.033266] _raw_spin_lock+0x21/0x30 [10681.035457] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10681.038000] ? do_raw_spin_unlock+0x75/0x190 [10681.039288] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10681.043180] ? do_wait_intr+0xf0/0xf0 [10681.049166] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10681.054660] mdc_close+0x53b/0xd70 [mdc] [10681.062045] lmv_close+0x18d/0x490 [lmv] [10681.066153] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10681.071949] ll_md_real_close+0xa5/0x2a0 [lustre] [10681.083691] ll_md_close+0x274/0x800 [lustre] [10681.086171] ll_file_release+0x128/0xc50 [lustre] [10681.089087] __fput+0x10d/0x440 [10681.091556] ____fput+0x12/0x20 [10681.093220] task_work_run+0xc6/0x110 [10681.095340] exit_to_usermode_loop+0x1dd/0x1f0 [10681.105163] do_syscall_64+0x3d6/0x440 [10681.110970] entry_SYSCALL_64_after_hwframe+0x49/0xae [10681.117716] RIP: 0033:0x7f1a63983cd8 [10681.119121] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10681.134107] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10681.143285] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10681.152426] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10681.160762] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10681.168192] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10681.175738] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10692.053659] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10692.059071] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10692.124264] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10692.140683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10692.151146] RIP: 0010:native_safe_halt+0x2e/0x50 [10692.156973] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10692.186856] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10692.197926] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10692.208305] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [10692.211153] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10692.217692] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10692.226081] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10692.241000] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10692.254957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10692.262435] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10692.270966] Call Trace: [10692.273285] [10692.274024] ? show_regs.cold.9+0x22/0x2f [10692.285004] ? watchdog_timer_fn.cold.6+0x62/0x10a [10692.287523] ? __hrtimer_run_queues+0x1c2/0x4d0 [10692.292756] ? watchdog+0x40/0x40 [10692.294403] ? hrtimer_interrupt+0x128/0x2c0 [10692.296806] ? smp_apic_timer_interrupt+0xa6/0x270 [10692.304068] ? apic_timer_interrupt+0xf/0x20 [10692.316008] [10692.319373] ? native_safe_halt+0x2e/0x50 [10692.327000] kvm_wait+0xcd/0xe0 [10692.328958] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10692.333882] do_raw_spin_lock+0x177/0x190 [10692.340897] _raw_spin_lock+0x21/0x30 [10692.347102] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10692.352028] ? do_raw_spin_unlock+0x75/0x190 [10692.370432] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10692.378979] ? do_wait_intr+0xf0/0xf0 [10692.380976] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10692.387762] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10692.389778] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10692.399764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10692.406989] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10692.409163] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10692.410933] mdc_intent_lock+0x353/0x7e0 [mdc] [10692.415147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10692.423023] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10692.428177] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10692.434109] lmv_intent_open+0x338/0x1150 [lmv] [10692.437184] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10692.439118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10692.442813] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10692.444789] ll_intent_lock+0xc5/0xaa0 [lustre] [10692.453057] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10692.462965] ll_lookup_it+0xe0e/0x2270 [lustre] [10692.469329] ? do_raw_spin_unlock+0x75/0x190 [10692.475718] ? _raw_spin_unlock+0x12/0x30 [10692.483267] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10692.494313] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10692.505060] ll_atomic_open+0x36a/0xd70 [lustre] [10692.515306] lookup_open+0x356/0xb40 [10692.516590] do_last+0x313/0xfc0 [10692.517819] ? nd_jump_root+0xe5/0x160 [10692.530507] ? path_init+0x437/0x520 [10692.538990] path_openat+0xf7/0x500 [10692.546552] ? do_raw_spin_unlock+0x75/0x190 [10692.559447] do_filp_open+0x99/0x140 [10692.568344] ? getname_flags+0x6e/0x330 [10692.576404] ? __check_object_size+0xff/0x256 [10692.585134] ? do_raw_spin_unlock+0x75/0x190 [10692.593106] ? _raw_spin_unlock+0x12/0x30 [10692.594594] do_sys_openat2+0x2b4/0x410 [10692.602311] do_sys_open+0x73/0xa0 [10692.604403] __x64_sys_openat+0x24/0x30 [10692.607638] do_syscall_64+0xc1/0x440 [10692.612479] entry_SYSCALL_64_after_hwframe+0x49/0xae [10692.616682] RIP: 0033:0x7f1a63983332 [10692.617963] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10692.646888] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10692.655891] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10692.675453] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10692.683682] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10692.692732] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10692.696768] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10696.162889] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8673s! [10696.166858] Showing busy workqueues and worker pools: [10696.169915] workqueue events: flags=0x0 [10696.172260] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10696.175590] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10696.180331] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10696.183085] pending: sec_gc_main [ptlrpc] [10696.188953] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10696.195644] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10696.200050] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10696.202661] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10696.206057] workqueue events_long: flags=0x0 [10696.207397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10696.210026] pending: lu_objects_flush [obdclass] [10696.212340] workqueue events_unbound: flags=0x2 [10696.213894] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10696.217122] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10696.221536] workqueue events_power_efficient: flags=0x80 [10696.223458] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10696.226648] pending: neigh_periodic_work, do_cache_clean [10696.228762] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10696.231829] pending: neigh_periodic_work, check_lifetime [10696.233951] workqueue mm_percpu_wq: flags=0x8 [10696.235805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10696.238834] pending: vmstat_update [10696.240084] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10696.242411] pending: vmstat_update [10696.243602] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10696.245902] pending: vmstat_update [10696.247206] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10696.250159] pending: vmstat_update [10696.252575] workqueue writeback: flags=0x4a [10696.254016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10696.256193] pending: wb_workfn, wb_workfn [10696.258324] workqueue xprtiod: flags=0xa [10696.259712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10696.262904] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10696.266795] workqueue ipv6_addrconf: flags=0x40008 [10696.268796] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10696.271788] pending: addrconf_verify_work [10696.273522] workqueue knbd0-recv: flags=0x1a [10696.274974] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10696.277769] in-flight: 647:recv_work [10696.279649] workqueue ptlrpc_pinger: flags=0x2 [10696.281982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10696.284907] pending: ptlrpc_pinger_main [ptlrpc] [10696.287344] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10686s workers=2 idle: 652 [10708.057028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10708.057639] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10708.058912] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10708.058920] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10708.058990] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10708.058997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10708.058999] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10708.059013] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10708.059018] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10708.059022] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10708.059024] RDX: 0000000000007260 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10708.059027] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10708.059029] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10708.059032] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10708.059036] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10708.059039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10708.059047] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10708.059055] Call Trace: [10708.059061] [10708.059076] ? show_regs.cold.9+0x22/0x2f [10708.059089] ? watchdog_timer_fn.cold.6+0x62/0x10a [10708.059096] ? __hrtimer_run_queues+0x1c2/0x4d0 [10708.059115] ? watchdog+0x40/0x40 [10708.059122] ? hrtimer_interrupt+0x128/0x2c0 [10708.059127] ? smp_apic_timer_interrupt+0xa6/0x270 [10708.059139] ? apic_timer_interrupt+0xf/0x20 [10708.059149] [10708.059152] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10708.059160] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10708.059168] do_raw_spin_lock+0x177/0x190 [10708.059173] _raw_spin_lock+0x21/0x30 [10708.059179] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10708.059557] obd_mod_free+0x46/0x140 [ptlrpc] [10708.059930] mdc_commit_open+0x91/0xb0 [mdc] [10708.059969] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10708.064186] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10708.064267] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10708.064273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10708.064276] RIP: 0010:native_safe_halt+0x2e/0x50 [10708.064288] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10708.064292] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10708.064297] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10708.064299] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10708.064301] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10708.064303] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10708.064306] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10708.061142] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10708.064309] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10708.065021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10708.065023] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10708.065029] Call Trace: [10708.065034] [10708.065036] ? show_regs.cold.9+0x22/0x2f [10708.065046] ? watchdog_timer_fn.cold.6+0x62/0x10a [10708.065050] ? __hrtimer_run_queues+0x1c2/0x4d0 [10708.065053] ? watchdog+0x40/0x40 [10708.064695] after_reply+0xe2c/0x1780 [ptlrpc] [10708.065056] ? hrtimer_interrupt+0x128/0x2c0 [10708.065187] ? smp_apic_timer_interrupt+0xa6/0x270 [10708.065193] ? apic_timer_interrupt+0xf/0x20 [10708.065198] [10708.065201] ? native_safe_halt+0x2e/0x50 [10708.065204] kvm_wait+0xcd/0xe0 [10708.065213] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10708.065220] do_raw_spin_lock+0x177/0x190 [10708.065224] _raw_spin_lock+0x21/0x30 [10708.065228] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10708.065365] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10708.066204] ? __next_timer_interrupt+0x160/0x160 [10708.066217] ptlrpcd+0xa82/0xe40 [ptlrpc] [10708.067202] ? do_wait_intr+0xf0/0xf0 [10708.067220] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10708.068434] kthread+0x1d1/0x200 [10708.068441] ? set_kthread_struct+0x70/0x70 [10708.068446] ret_from_fork+0x1f/0x30 [10708.069562] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10708.069631] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10708.069638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10708.069641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10708.069653] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10708.069657] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10708.069662] RAX: 0000000000002d59 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10708.069665] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10708.069667] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10708.069669] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10708.069672] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10708.069675] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10708.069678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10708.069680] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10708.069689] Call Trace: [10708.069694] [10708.069698] ? show_regs.cold.9+0x22/0x2f [10708.069707] ? watchdog_timer_fn.cold.6+0x62/0x10a [10708.069714] ? __hrtimer_run_queues+0x1c2/0x4d0 [10708.069718] ? watchdog+0x40/0x40 [10708.069722] ? hrtimer_interrupt+0x128/0x2c0 [10708.069727] ? smp_apic_timer_interrupt+0xa6/0x270 [10708.069735] ? apic_timer_interrupt+0xf/0x20 [10708.069740] [10708.069742] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10708.069749] do_raw_spin_lock+0x177/0x190 [10708.069754] _raw_spin_lock+0x21/0x30 [10708.069758] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10708.070022] ? do_raw_spin_unlock+0x75/0x190 [10708.070029] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10708.070301] ? do_wait_intr+0xf0/0xf0 [10708.070309] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10708.070705] mdc_close+0x53b/0xd70 [mdc] [10708.070750] lmv_close+0x18d/0x490 [lmv] [10708.070778] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10708.070953] ll_md_real_close+0xa5/0x2a0 [lustre] [10708.071034] ll_md_close+0x274/0x800 [lustre] [10708.071236] ll_file_release+0x128/0xc50 [lustre] [10708.071397] __fput+0x10d/0x440 [10708.071407] ____fput+0x12/0x20 [10708.071412] task_work_run+0xc6/0x110 [10708.071420] exit_to_usermode_loop+0x1dd/0x1f0 [10708.071428] do_syscall_64+0x3d6/0x440 [10708.071432] entry_SYSCALL_64_after_hwframe+0x49/0xae [10708.071511] RIP: 0033:0x7f1a63983cd8 [10708.071520] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10708.071524] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10708.071529] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10708.071532] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10708.071534] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10708.071536] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10708.071538] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10708.924671] ? do_raw_spin_unlock+0x75/0x190 [10708.927167] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10708.932781] ? do_wait_intr+0xf0/0xf0 [10708.936449] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10708.941696] mdc_close+0x53b/0xd70 [mdc] [10708.944741] lmv_close+0x18d/0x490 [lmv] [10708.947435] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10708.957251] ll_md_real_close+0xa5/0x2a0 [lustre] [10708.960242] ll_md_close+0x274/0x800 [lustre] [10708.963611] ll_file_release+0x128/0xc50 [lustre] [10708.966242] __fput+0x10d/0x440 [10708.969182] ____fput+0x12/0x20 [10708.970526] task_work_run+0xc6/0x110 [10708.974605] exit_to_usermode_loop+0x1dd/0x1f0 [10708.976500] do_syscall_64+0x3d6/0x440 [10708.980288] entry_SYSCALL_64_after_hwframe+0x49/0xae [10708.986882] RIP: 0033:0x7f1a63983cd8 [10708.988182] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10709.002138] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10709.005779] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10709.008273] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10709.013725] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10709.017633] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10709.027909] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10722.645120] rcu: INFO: rcu_sched self-detected stall on CPU [10722.647994] rcu: 3-...!: (5484997 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [10722.653106] (t=8700245 jiffies g=269633 q=3141) [10722.655744] rcu: rcu_sched kthread starved for 8699893 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [10722.660822] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10722.673628] rcu: RCU grace-period kthread stack dump: [10722.681872] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10722.690906] Call Trace: [10722.696038] __schedule+0x351/0xcb0 [10722.698104] schedule+0xc0/0x180 [10722.703947] schedule_timeout+0xb4/0x190 [10722.705694] ? do_raw_spin_unlock+0x75/0x190 [10722.711663] ? __next_timer_interrupt+0x160/0x160 [10722.715896] ? prepare_to_swait_event+0xf3/0x140 [10722.726898] rcu_gp_kthread+0x8dd/0x15b0 [10722.730704] ? force_qs_rnp+0x2c0/0x2c0 [10722.735259] kthread+0x1d1/0x200 [10722.738293] ? set_kthread_struct+0x70/0x70 [10722.741177] ret_from_fork+0x1f/0x30 [10722.742509] rcu: Stack dump where RCU GP kthread last ran: [10722.745697] Sending NMI from CPU 3 to CPUs 0: [10722.754315] NMI backtrace for cpu 0 [10722.754323] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10722.754325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10722.754327] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10722.754329] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10722.754331] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 [10722.754334] RAX: 0000000000004d14 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10722.754335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10722.754336] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10722.754338] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10722.754339] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10722.754341] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10722.754342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10722.754344] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10722.754345] Call Trace: [10722.754346] [10722.754347] ? show_regs.cold.9+0x22/0x2f [10722.754348] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10722.754349] ? nmi_cpu_backtrace_handler+0x15/0x40 [10722.754350] ? nmi_handle+0xbd/0x210 [10722.754352] ? default_do_nmi+0x4b/0x1b0 [10722.754353] ? do_nmi+0x280/0x380 [10722.754354] ? end_repeat_nmi+0x16/0x1f [10722.754355] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10722.754356] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10722.754357] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10722.754358] [10722.754359] do_raw_spin_lock+0x177/0x190 [10722.754360] _raw_spin_lock+0x21/0x30 [10722.754361] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10722.754362] ? do_raw_spin_unlock+0x75/0x190 [10722.754363] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10722.754364] ? do_wait_intr+0xf0/0xf0 [10722.754365] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10722.754366] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10722.754367] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10722.754368] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10722.754369] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10722.754370] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10722.754371] mdc_intent_lock+0x353/0x7e0 [mdc] [10722.754373] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10722.754374] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10722.754375] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10722.754376] lmv_intent_open+0x338/0x1150 [lmv] [10722.754377] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10722.754378] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10722.754380] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10722.754380] ll_intent_lock+0xc5/0xaa0 [lustre] [10722.754381] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10722.754383] ll_lookup_it+0xe0e/0x2270 [lustre] [10722.754384] ? do_raw_spin_unlock+0x75/0x190 [10722.754385] ? _raw_spin_unlock+0x12/0x30 [10722.754387] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10722.754388] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10722.754389] ll_atomic_open+0x36a/0xd70 [lustre] [10722.754390] lookup_open+0x356/0xb40 [10722.754391] do_last+0x313/0xfc0 [10722.754392] ? nd_jump_root+0xe5/0x160 [10722.754393] ? path_init+0x437/0x520 [10722.754394] path_openat+0xf7/0x500 [10722.754395] ? do_raw_spin_unlock+0x75/0x190 [10722.754396] do_filp_open+0x99/0x140 [10722.754398] ? getname_flags+0x6e/0x330 [10722.754399] ? __check_object_size+0xff/0x256 [10722.754400] ? do_raw_spin_unlock+0x75/0x190 [10722.754401] ? _raw_spin_unlock+0x12/0x30 [10722.754403] do_sys_openat2+0x2b4/0x410 [10722.754404] do_sys_open+0x73/0xa0 [10722.754405] __x64_sys_openat+0x24/0x30 [10722.754406] do_syscall_64+0xc1/0x440 [10722.754407] entry_SYSCALL_64_after_hwframe+0x49/0xae [10722.754408] RIP: 0033:0x7f1a63983332 [10722.754409] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10722.754412] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10722.754415] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10722.754416] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10722.754418] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10722.754420] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10722.754421] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10722.759411] NMI backtrace for cpu 3 [10723.012508] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10723.018984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10723.024684] Call Trace: [10723.026230] [10723.027500] ? dump_stack+0xbb/0x10e [10723.029134] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10723.032109] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10723.034615] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10723.040583] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10723.043636] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10723.047746] ? rcu_dump_cpu_stacks+0x11b/0x171 [10723.050853] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10723.055396] ? update_process_times+0x6d/0xc0 [10723.059802] ? tick_sched_handle.isra.18+0x27/0x90 [10723.062004] ? tick_sched_timer+0x53/0xb0 [10723.064397] ? __hrtimer_run_queues+0x1c2/0x4d0 [10723.066580] ? tick_sched_handle.isra.18+0x90/0x90 [10723.069381] ? hrtimer_interrupt+0x128/0x2c0 [10723.072278] ? smp_apic_timer_interrupt+0xa6/0x270 [10723.074623] ? apic_timer_interrupt+0xf/0x20 [10723.076938] [10723.078601] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10723.081845] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10723.084895] ? do_raw_spin_lock+0x177/0x190 [10723.087460] ? _raw_spin_lock+0x21/0x30 [10723.089700] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10723.094111] ? obd_mod_free+0x46/0x140 [ptlrpc] [10723.097415] ? mdc_commit_open+0x91/0xb0 [mdc] [10723.100403] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10723.104303] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10723.108522] ? after_reply+0xe2c/0x1780 [ptlrpc] [10723.112222] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10723.115312] ? __next_timer_interrupt+0x160/0x160 [10723.117790] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10723.122717] ? do_wait_intr+0xf0/0xf0 [10723.125598] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10723.140300] ? kthread+0x1d1/0x200 [10723.142662] ? set_kthread_struct+0x70/0x70 [10723.147789] ? ret_from_fork+0x1f/0x30 [10726.369046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8703s! [10726.372639] Showing busy workqueues and worker pools: [10726.374172] workqueue events: flags=0x0 [10726.375617] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10726.379429] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10726.382703] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10726.387409] pending: sec_gc_main [ptlrpc] [10726.389541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10726.392235] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10726.396844] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10726.399595] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10726.403602] workqueue events_long: flags=0x0 [10726.405826] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10726.408629] pending: lu_objects_flush [obdclass] [10726.411127] workqueue events_unbound: flags=0x2 [10726.412994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10726.415349] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10726.418565] workqueue events_power_efficient: flags=0x80 [10726.420250] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10726.422392] pending: neigh_periodic_work, do_cache_clean [10726.424380] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10726.427097] pending: neigh_periodic_work, check_lifetime [10726.429077] workqueue mm_percpu_wq: flags=0x8 [10726.430949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10726.433282] pending: vmstat_update [10726.434626] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10726.438021] pending: vmstat_update [10726.439593] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10726.442633] pending: vmstat_update [10726.444776] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10726.447750] pending: vmstat_update [10726.449551] workqueue writeback: flags=0x4a [10726.451422] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10726.454159] pending: wb_workfn, wb_workfn [10726.455903] workqueue xprtiod: flags=0xa [10726.457859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10726.460619] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10726.464472] workqueue ipv6_addrconf: flags=0x40008 [10726.466652] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10726.469877] pending: addrconf_verify_work [10726.471700] workqueue knbd0-recv: flags=0x1a [10726.473567] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10726.476176] in-flight: 647:recv_work [10726.478035] workqueue ptlrpc_pinger: flags=0x2 [10726.479897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10726.481924] pending: ptlrpc_pinger_main [ptlrpc] [10726.484379] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10716s workers=2 idle: 652 [10736.056120] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10736.060869] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10736.062391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10736.062535] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10736.062542] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10736.062545] RIP: 0010:native_safe_halt+0x2e/0x50 [10736.062558] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10736.062562] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10736.062567] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10736.062570] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10736.062572] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10736.062574] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10736.062577] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10736.062580] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10736.062584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10736.062586] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10736.062596] Call Trace: [10736.062601] [10736.062607] ? show_regs.cold.9+0x22/0x2f [10736.069396] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10736.069541] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10736.069554] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10736.069557] RIP: 0010:native_safe_halt+0x2e/0x50 [10736.069574] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10736.069581] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10736.069586] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10736.069588] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [10736.069590] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10736.069592] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10736.069598] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10736.069600] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10736.069603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10736.069605] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10736.069674] Call Trace: [10736.069742] [10736.069751] ? show_regs.cold.9+0x22/0x2f [10736.069766] ? watchdog_timer_fn.cold.6+0x62/0x10a [10736.069773] ? __hrtimer_run_queues+0x1c2/0x4d0 [10736.069777] ? watchdog+0x40/0x40 [10736.069784] ? hrtimer_interrupt+0x128/0x2c0 [10736.069859] ? smp_apic_timer_interrupt+0xa6/0x270 [10736.069866] ? apic_timer_interrupt+0xf/0x20 [10736.069875] [10736.069878] ? native_safe_halt+0x2e/0x50 [10736.069882] kvm_wait+0xcd/0xe0 [10736.069890] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10736.069903] do_raw_spin_lock+0x177/0x190 [10736.069907] _raw_spin_lock+0x21/0x30 [10736.069912] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10736.070511] ? do_raw_spin_unlock+0x75/0x190 [10736.070518] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10736.071144] ? do_wait_intr+0xf0/0xf0 [10736.071152] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10736.071499] mdc_close+0x53b/0xd70 [mdc] [10736.071545] lmv_close+0x18d/0x490 [lmv] [10736.071579] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10736.071940] ll_md_real_close+0xa5/0x2a0 [lustre] [10736.072247] ll_md_close+0x274/0x800 [lustre] [10736.072399] ll_file_release+0x128/0xc50 [lustre] [10736.072566] __fput+0x10d/0x440 [10736.072576] ____fput+0x12/0x20 [10736.072581] task_work_run+0xc6/0x110 [10736.072588] exit_to_usermode_loop+0x1dd/0x1f0 [10736.072595] do_syscall_64+0x3d6/0x440 [10736.072600] entry_SYSCALL_64_after_hwframe+0x49/0xae [10736.072607] RIP: 0033:0x7f1a63983cd8 [10736.072679] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10736.072738] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10736.072744] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10736.072746] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10736.072748] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10736.072751] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10736.072753] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10736.550890] ? watchdog_timer_fn.cold.6+0x62/0x10a [10736.557690] ? __hrtimer_run_queues+0x1c2/0x4d0 [10736.560664] ? watchdog+0x40/0x40 [10736.561639] ? hrtimer_interrupt+0x128/0x2c0 [10736.567155] ? smp_apic_timer_interrupt+0xa6/0x270 [10736.568908] ? apic_timer_interrupt+0xf/0x20 [10736.573035] [10736.576738] ? native_safe_halt+0x2e/0x50 [10736.582787] kvm_wait+0xcd/0xe0 [10736.585751] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10736.594317] do_raw_spin_lock+0x177/0x190 [10736.595914] _raw_spin_lock+0x21/0x30 [10736.598560] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10736.607731] ? do_raw_spin_unlock+0x75/0x190 [10736.610748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10736.617480] ? do_wait_intr+0xf0/0xf0 [10736.618806] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10736.620766] mdc_close+0x53b/0xd70 [mdc] [10736.623147] lmv_close+0x18d/0x490 [lmv] [10736.625327] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10736.627743] ll_md_real_close+0xa5/0x2a0 [lustre] [10736.639384] ll_md_close+0x274/0x800 [lustre] [10736.649651] ll_file_release+0x128/0xc50 [lustre] [10736.652736] __fput+0x10d/0x440 [10736.654340] ____fput+0x12/0x20 [10736.655703] task_work_run+0xc6/0x110 [10736.661529] exit_to_usermode_loop+0x1dd/0x1f0 [10736.664301] do_syscall_64+0x3d6/0x440 [10736.667335] entry_SYSCALL_64_after_hwframe+0x49/0xae [10736.673126] RIP: 0033:0x7f1a63983cd8 [10736.679444] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10736.696591] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10736.709333] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10736.718185] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10736.731321] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10736.736996] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10736.741271] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10748.059789] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10748.065384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10748.093188] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10748.098443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10748.103238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10748.106513] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10748.116086] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10748.119384] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10748.122606] RDX: 0000000000000862 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10748.126632] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10748.130062] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10748.133818] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10748.141643] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10748.149241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10748.155264] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10748.159356] Call Trace: [10748.161051] [10748.162258] ? show_regs.cold.9+0x22/0x2f [10748.164608] ? watchdog_timer_fn.cold.6+0x62/0x10a [10748.167066] ? __hrtimer_run_queues+0x1c2/0x4d0 [10748.171041] ? watchdog+0x40/0x40 [10748.172561] ? hrtimer_interrupt+0x128/0x2c0 [10748.175310] ? smp_apic_timer_interrupt+0xa6/0x270 [10748.177427] ? apic_timer_interrupt+0xf/0x20 [10748.179471] [10748.180285] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10748.183166] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10748.185962] do_raw_spin_lock+0x177/0x190 [10748.187863] _raw_spin_lock+0x21/0x30 [10748.190088] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10748.193136] obd_mod_free+0x46/0x140 [ptlrpc] [10748.196025] mdc_commit_open+0x91/0xb0 [mdc] [10748.198464] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10748.202081] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10748.205530] after_reply+0xe2c/0x1780 [ptlrpc] [10748.210139] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10748.213016] ? __next_timer_interrupt+0x160/0x160 [10748.216384] ptlrpcd+0xa82/0xe40 [ptlrpc] [10748.219093] ? do_wait_intr+0xf0/0xf0 [10748.220893] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10748.224165] kthread+0x1d1/0x200 [10748.226143] ? set_kthread_struct+0x70/0x70 [10748.228535] ret_from_fork+0x1f/0x30 [10752.053022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10752.055994] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10752.076266] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10752.080326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10752.084006] RIP: 0010:native_safe_halt+0x2e/0x50 [10752.085862] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10752.092458] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10752.095204] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10752.097796] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [10752.100571] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10752.103053] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10752.105576] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10752.108121] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10752.110997] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10752.112976] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10752.115366] Call Trace: [10752.116252] [10752.117010] ? show_regs.cold.9+0x22/0x2f [10752.118554] ? watchdog_timer_fn.cold.6+0x62/0x10a [10752.120342] ? __hrtimer_run_queues+0x1c2/0x4d0 [10752.122049] ? watchdog+0x40/0x40 [10752.123219] ? hrtimer_interrupt+0x128/0x2c0 [10752.124773] ? smp_apic_timer_interrupt+0xa6/0x270 [10752.126358] ? apic_timer_interrupt+0xf/0x20 [10752.127964] [10752.128804] ? native_safe_halt+0x2e/0x50 [10752.130176] kvm_wait+0xcd/0xe0 [10752.131304] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10752.133152] do_raw_spin_lock+0x177/0x190 [10752.134954] _raw_spin_lock+0x21/0x30 [10752.136183] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10752.138333] ? do_raw_spin_unlock+0x75/0x190 [10752.139894] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10752.141759] ? do_wait_intr+0xf0/0xf0 [10752.143051] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10752.145072] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10752.147168] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10752.150036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10752.152132] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10752.154283] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10752.156210] mdc_intent_lock+0x353/0x7e0 [mdc] [10752.157989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10752.159929] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10752.162291] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10752.164374] lmv_intent_open+0x338/0x1150 [lmv] [10752.166140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10752.168148] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10752.169992] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10752.171873] ll_intent_lock+0xc5/0xaa0 [lustre] [10752.173858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10752.176028] ll_lookup_it+0xe0e/0x2270 [lustre] [10752.177955] ? do_raw_spin_unlock+0x75/0x190 [10752.179583] ? _raw_spin_unlock+0x12/0x30 [10752.181184] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10752.183158] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10752.185296] ll_atomic_open+0x36a/0xd70 [lustre] [10752.187151] lookup_open+0x356/0xb40 [10752.188607] do_last+0x313/0xfc0 [10752.189931] ? nd_jump_root+0xe5/0x160 [10752.191375] ? path_init+0x437/0x520 [10752.192851] path_openat+0xf7/0x500 [10752.194071] ? do_raw_spin_unlock+0x75/0x190 [10752.195600] do_filp_open+0x99/0x140 [10752.196992] ? getname_flags+0x6e/0x330 [10752.198287] ? __check_object_size+0xff/0x256 [10752.200084] ? do_raw_spin_unlock+0x75/0x190 [10752.202546] ? _raw_spin_unlock+0x12/0x30 [10752.203908] do_sys_openat2+0x2b4/0x410 [10752.205451] do_sys_open+0x73/0xa0 [10752.206832] __x64_sys_openat+0x24/0x30 [10752.208072] do_syscall_64+0xc1/0x440 [10752.209569] entry_SYSCALL_64_after_hwframe+0x49/0xae [10752.211403] RIP: 0033:0x7f1a63983332 [10752.212799] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10752.219411] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10752.222529] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10752.225252] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10752.228454] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10752.231058] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10752.233933] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10756.576815] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8733s! [10756.582172] Showing busy workqueues and worker pools: [10756.585399] workqueue events: flags=0x0 [10756.589574] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10756.592906] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10756.596376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10756.599622] pending: sec_gc_main [ptlrpc] [10756.602198] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10756.605726] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10756.611238] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10756.614483] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10756.619539] workqueue events_long: flags=0x0 [10756.621429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10756.624495] pending: lu_objects_flush [obdclass] [10756.627065] workqueue events_unbound: flags=0x2 [10756.629648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10756.632896] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10756.638738] workqueue events_power_efficient: flags=0x80 [10756.641405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10756.644884] pending: neigh_periodic_work, do_cache_clean [10756.647851] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10756.650945] pending: neigh_periodic_work, check_lifetime [10756.653770] workqueue mm_percpu_wq: flags=0x8 [10756.656065] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10756.659502] pending: vmstat_update [10756.661411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10756.664301] pending: vmstat_update [10756.666154] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10756.669236] pending: vmstat_update [10756.671052] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10756.674203] pending: vmstat_update [10756.676245] workqueue writeback: flags=0x4a [10756.678121] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10756.681186] pending: wb_workfn, wb_workfn [10756.683416] workqueue xprtiod: flags=0xa [10756.685047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10756.688256] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10756.692207] workqueue ipv6_addrconf: flags=0x40008 [10756.694487] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10756.697263] pending: addrconf_verify_work [10756.699035] workqueue knbd0-recv: flags=0x1a [10756.700788] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10756.703439] in-flight: 647:recv_work [10756.705267] workqueue ptlrpc_pinger: flags=0x2 [10756.707406] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10756.709804] pending: ptlrpc_pinger_main [ptlrpc] [10756.712539] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10746s workers=2 idle: 652 [10764.057011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10764.058089] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10764.059753] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10764.059830] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10764.059838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10764.059842] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10764.059857] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10764.059861] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10764.059865] RAX: 0000000000003e6f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10764.059867] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10764.059870] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10764.059872] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10764.059874] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10764.059878] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10764.059881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10764.059883] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10764.059939] Call Trace: [10764.059945] [10764.059950] ? show_regs.cold.9+0x22/0x2f [10764.059959] ? watchdog_timer_fn.cold.6+0x62/0x10a [10764.059965] ? __hrtimer_run_queues+0x1c2/0x4d0 [10764.059969] ? watchdog+0x40/0x40 [10764.059973] ? hrtimer_interrupt+0x128/0x2c0 [10764.059976] ? smp_apic_timer_interrupt+0xa6/0x270 [10764.059984] ? apic_timer_interrupt+0xf/0x20 [10764.059988] [10764.059990] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10764.059996] do_raw_spin_lock+0x177/0x190 [10764.060000] _raw_spin_lock+0x21/0x30 [10764.060004] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10764.060418] ? do_raw_spin_unlock+0x75/0x190 [10764.060428] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10764.061231] ? do_wait_intr+0xf0/0xf0 [10764.061242] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10764.061693] mdc_close+0x53b/0xd70 [mdc] [10764.061734] lmv_close+0x18d/0x490 [lmv] [10764.061765] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10764.061876] ll_md_real_close+0xa5/0x2a0 [lustre] [10764.061954] ll_md_close+0x274/0x800 [lustre] [10764.062034] ll_file_release+0x128/0xc50 [lustre] [10764.062114] __fput+0x10d/0x440 [10764.062124] ____fput+0x12/0x20 [10764.062128] task_work_run+0xc6/0x110 [10764.062135] exit_to_usermode_loop+0x1dd/0x1f0 [10764.062142] do_syscall_64+0x3d6/0x440 [10764.062147] entry_SYSCALL_64_after_hwframe+0x49/0xae [10764.062155] RIP: 0033:0x7f1a63983cd8 [10764.062161] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10764.062165] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10764.062169] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10764.062171] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10764.062174] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10764.062176] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10764.062178] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10764.423917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10764.463578] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10764.476487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10764.483683] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10764.488985] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10764.504011] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10764.515930] RAX: 000000000000628c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10764.519036] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10764.521314] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10764.527322] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10764.533264] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10764.539287] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10764.546180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10764.555917] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10764.559626] Call Trace: [10764.560502] [10764.563914] ? show_regs.cold.9+0x22/0x2f [10764.565430] ? watchdog_timer_fn.cold.6+0x62/0x10a [10764.573799] ? __hrtimer_run_queues+0x1c2/0x4d0 [10764.583057] ? watchdog+0x40/0x40 [10764.587054] ? hrtimer_interrupt+0x128/0x2c0 [10764.593158] ? smp_apic_timer_interrupt+0xa6/0x270 [10764.602233] ? apic_timer_interrupt+0xf/0x20 [10764.608259] [10764.610872] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10764.620742] do_raw_spin_lock+0x177/0x190 [10764.629280] _raw_spin_lock+0x21/0x30 [10764.639566] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10764.646495] ? do_raw_spin_unlock+0x75/0x190 [10764.648581] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10764.657262] ? do_wait_intr+0xf0/0xf0 [10764.662642] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10764.668270] mdc_close+0x53b/0xd70 [mdc] [10764.673474] lmv_close+0x18d/0x490 [lmv] [10764.680469] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10764.687868] ll_md_real_close+0xa5/0x2a0 [lustre] [10764.691502] ll_md_close+0x274/0x800 [lustre] [10764.712140] ll_file_release+0x128/0xc50 [lustre] [10764.718767] __fput+0x10d/0x440 [10764.721027] ____fput+0x12/0x20 [10764.728438] task_work_run+0xc6/0x110 [10764.730293] exit_to_usermode_loop+0x1dd/0x1f0 [10764.739782] do_syscall_64+0x3d6/0x440 [10764.745434] entry_SYSCALL_64_after_hwframe+0x49/0xae [10764.751970] RIP: 0033:0x7f1a63983cd8 [10764.753263] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10764.775298] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10764.787327] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10764.799726] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10764.810546] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10764.819165] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10764.831187] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10776.058032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10776.061488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10776.086128] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10776.092329] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10776.096897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10776.100285] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10776.109870] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10776.113703] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10776.117709] RDX: 0000000000007515 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10776.121080] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10776.125329] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10776.128738] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10776.132728] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10776.136457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10776.139429] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10776.142850] Call Trace: [10776.144283] [10776.145187] ? show_regs.cold.9+0x22/0x2f [10776.147075] ? watchdog_timer_fn.cold.6+0x62/0x10a [10776.149184] ? __hrtimer_run_queues+0x1c2/0x4d0 [10776.151385] ? watchdog+0x40/0x40 [10776.153385] ? hrtimer_interrupt+0x128/0x2c0 [10776.155717] ? smp_apic_timer_interrupt+0xa6/0x270 [10776.158108] ? apic_timer_interrupt+0xf/0x20 [10776.160127] [10776.161352] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10776.164597] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10776.167548] do_raw_spin_lock+0x177/0x190 [10776.169582] _raw_spin_lock+0x21/0x30 [10776.171860] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10776.175287] obd_mod_free+0x46/0x140 [ptlrpc] [10776.178108] mdc_commit_open+0x91/0xb0 [mdc] [10776.180221] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10776.183462] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10776.185877] after_reply+0xe2c/0x1780 [ptlrpc] [10776.188336] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10776.190119] ? __next_timer_interrupt+0x160/0x160 [10776.192077] ptlrpcd+0xa82/0xe40 [ptlrpc] [10776.193930] ? do_wait_intr+0xf0/0xf0 [10776.195116] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10776.197248] kthread+0x1d1/0x200 [10776.198602] ? set_kthread_struct+0x70/0x70 [10776.200562] ret_from_fork+0x1f/0x30 [10784.058900] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10784.070582] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10784.108090] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10784.116500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10784.128466] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10784.137964] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10784.159724] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10784.163910] RAX: 000000000000538f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10784.172640] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10784.181234] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10784.189008] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10784.197004] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10784.199504] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10784.206643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10784.210703] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10784.219418] Call Trace: [10784.223613] [10784.225102] ? show_regs.cold.9+0x22/0x2f [10784.233815] ? watchdog_timer_fn.cold.6+0x62/0x10a [10784.248005] ? __hrtimer_run_queues+0x1c2/0x4d0 [10784.249591] ? watchdog+0x40/0x40 [10784.256516] ? hrtimer_interrupt+0x128/0x2c0 [10784.261510] ? smp_apic_timer_interrupt+0xa6/0x270 [10784.266473] ? apic_timer_interrupt+0xf/0x20 [10784.275663] [10784.276496] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10784.278204] do_raw_spin_lock+0x177/0x190 [10784.288113] _raw_spin_lock+0x21/0x30 [10784.290063] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10784.293108] ? do_raw_spin_unlock+0x75/0x190 [10784.295003] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10784.298002] ? do_wait_intr+0xf0/0xf0 [10784.299629] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10784.302308] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10784.304625] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10784.307642] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10784.310280] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10784.313188] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10784.315081] mdc_intent_lock+0x353/0x7e0 [mdc] [10784.317084] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10784.319540] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10784.322213] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10784.324561] lmv_intent_open+0x338/0x1150 [lmv] [10784.326474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10784.328640] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10784.330872] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10784.332723] ll_intent_lock+0xc5/0xaa0 [lustre] [10784.335034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10784.337476] ll_lookup_it+0xe0e/0x2270 [lustre] [10784.339306] ? do_raw_spin_unlock+0x75/0x190 [10784.340893] ? _raw_spin_unlock+0x12/0x30 [10784.342561] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10784.344882] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10784.347013] ll_atomic_open+0x36a/0xd70 [lustre] [10784.354081] lookup_open+0x356/0xb40 [10784.356418] do_last+0x313/0xfc0 [10784.358176] ? nd_jump_root+0xe5/0x160 [10784.359940] ? path_init+0x437/0x520 [10784.362962] path_openat+0xf7/0x500 [10784.367442] ? do_raw_spin_unlock+0x75/0x190 [10784.368862] do_filp_open+0x99/0x140 [10784.370004] ? getname_flags+0x6e/0x330 [10784.372227] ? __check_object_size+0xff/0x256 [10784.376091] ? do_raw_spin_unlock+0x75/0x190 [10784.379388] ? _raw_spin_unlock+0x12/0x30 [10784.381110] do_sys_openat2+0x2b4/0x410 [10784.387548] do_sys_open+0x73/0xa0 [10784.392201] __x64_sys_openat+0x24/0x30 [10784.394463] do_syscall_64+0xc1/0x440 [10784.396722] entry_SYSCALL_64_after_hwframe+0x49/0xae [10784.398768] RIP: 0033:0x7f1a63983332 [10784.401516] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10784.411226] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10784.416479] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10784.419989] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10784.423497] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10784.427819] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10784.430291] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10786.797503] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8763s! [10786.814462] Showing busy workqueues and worker pools: [10786.819874] workqueue events: flags=0x0 [10786.824059] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10786.836878] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10786.856717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10786.878278] pending: sec_gc_main [ptlrpc] [10786.881949] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10786.894992] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10786.915773] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10786.918249] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10786.938022] workqueue events_long: flags=0x0 [10786.943621] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10786.950498] pending: lu_objects_flush [obdclass] [10786.956638] workqueue events_unbound: flags=0x2 [10786.961742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10786.966666] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10786.973250] workqueue events_power_efficient: flags=0x80 [10786.980691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10786.991439] pending: neigh_periodic_work, do_cache_clean [10786.998036] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10787.008253] pending: neigh_periodic_work, check_lifetime [10787.010907] workqueue mm_percpu_wq: flags=0x8 [10787.017840] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10787.023339] pending: vmstat_update [10787.027001] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10787.033635] pending: vmstat_update [10787.036714] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10787.042122] pending: vmstat_update [10787.044739] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10787.053036] pending: vmstat_update [10787.059712] workqueue writeback: flags=0x4a [10787.064413] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10787.073883] pending: wb_workfn, wb_workfn [10787.077029] workqueue xprtiod: flags=0xa [10787.081628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10787.085227] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10787.095633] workqueue ipv6_addrconf: flags=0x40008 [10787.098830] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10787.103915] pending: addrconf_verify_work [10787.107307] workqueue knbd0-recv: flags=0x1a [10787.109887] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10787.120720] in-flight: 647:recv_work [10787.140107] workqueue ptlrpc_pinger: flags=0x2 [10787.154962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10787.165076] pending: ptlrpc_pinger_main [ptlrpc] [10787.181492] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10777s workers=2 idle: 652 [10792.057008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10792.061307] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10792.062770] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10792.062844] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10792.062852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10792.062855] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10792.062869] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10792.062873] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10792.062877] RAX: 0000000000006c68 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10792.062880] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10792.062882] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10792.062884] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10792.062887] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10792.062890] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10792.062893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10792.062895] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10792.062905] Call Trace: [10792.062910] [10792.062917] ? show_regs.cold.9+0x22/0x2f [10792.062925] ? watchdog_timer_fn.cold.6+0x62/0x10a [10792.062932] ? __hrtimer_run_queues+0x1c2/0x4d0 [10792.062936] ? watchdog+0x40/0x40 [10792.062940] ? hrtimer_interrupt+0x128/0x2c0 [10792.062944] ? smp_apic_timer_interrupt+0xa6/0x270 [10792.062952] ? apic_timer_interrupt+0xf/0x20 [10792.062957] [10792.062959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10792.062965] do_raw_spin_lock+0x177/0x190 [10792.062970] _raw_spin_lock+0x21/0x30 [10792.062975] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10792.063273] ? do_raw_spin_unlock+0x75/0x190 [10792.063279] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10792.063480] ? do_wait_intr+0xf0/0xf0 [10792.063487] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10792.063679] mdc_close+0x53b/0xd70 [mdc] [10792.063717] lmv_close+0x18d/0x490 [lmv] [10792.063745] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10792.063861] ll_md_real_close+0xa5/0x2a0 [lustre] [10792.063940] ll_md_close+0x274/0x800 [lustre] [10792.064023] ll_file_release+0x128/0xc50 [lustre] [10792.064108] __fput+0x10d/0x440 [10792.064117] ____fput+0x12/0x20 [10792.064122] task_work_run+0xc6/0x110 [10792.064129] exit_to_usermode_loop+0x1dd/0x1f0 [10792.064136] do_syscall_64+0x3d6/0x440 [10792.064141] entry_SYSCALL_64_after_hwframe+0x49/0xae [10792.064148] RIP: 0033:0x7f1a63983cd8 [10792.064154] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10792.064157] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10792.064162] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10792.064164] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10792.064166] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10792.064168] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10792.064170] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10792.263417] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10792.294067] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10792.299911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10792.307850] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10792.312875] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10792.329605] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10792.337828] RAX: 0000000000007d13 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10792.343001] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10792.346736] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10792.350437] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10792.356668] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10792.362784] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10792.371304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10792.375679] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10792.380699] Call Trace: [10792.383975] [10792.385580] ? show_regs.cold.9+0x22/0x2f [10792.387795] ? watchdog_timer_fn.cold.6+0x62/0x10a [10792.391009] ? __hrtimer_run_queues+0x1c2/0x4d0 [10792.393010] ? watchdog+0x40/0x40 [10792.395856] ? hrtimer_interrupt+0x128/0x2c0 [10792.399837] ? smp_apic_timer_interrupt+0xa6/0x270 [10792.404000] ? apic_timer_interrupt+0xf/0x20 [10792.408716] [10792.410886] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10792.414877] do_raw_spin_lock+0x177/0x190 [10792.416874] _raw_spin_lock+0x21/0x30 [10792.418999] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10792.422511] ? do_raw_spin_unlock+0x75/0x190 [10792.426493] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10792.431370] ? do_wait_intr+0xf0/0xf0 [10792.433777] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10792.439771] mdc_close+0x53b/0xd70 [mdc] [10792.444345] lmv_close+0x18d/0x490 [lmv] [10792.447390] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10792.451072] ll_md_real_close+0xa5/0x2a0 [lustre] [10792.453814] ll_md_close+0x274/0x800 [lustre] [10792.456392] ll_file_release+0x128/0xc50 [lustre] [10792.458505] __fput+0x10d/0x440 [10792.461401] ____fput+0x12/0x20 [10792.463912] task_work_run+0xc6/0x110 [10792.466601] exit_to_usermode_loop+0x1dd/0x1f0 [10792.468063] do_syscall_64+0x3d6/0x440 [10792.470868] entry_SYSCALL_64_after_hwframe+0x49/0xae [10792.474698] RIP: 0033:0x7f1a63983cd8 [10792.480498] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10792.493191] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10792.500136] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10792.507729] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10792.512803] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10792.517911] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10792.521956] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10804.058023] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [10804.060766] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10804.078365] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10804.082181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10804.085520] RIP: 0010:native_safe_halt+0x2e/0x50 [10804.087291] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10804.094216] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10804.097074] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10804.105105] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10804.108515] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10804.111409] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10804.113944] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10804.117749] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10804.120398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10804.122729] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10804.125377] Call Trace: [10804.126551] [10804.127481] ? show_regs.cold.9+0x22/0x2f [10804.130077] ? watchdog_timer_fn.cold.6+0x62/0x10a [10804.131975] ? __hrtimer_run_queues+0x1c2/0x4d0 [10804.134438] ? watchdog+0x40/0x40 [10804.136280] ? hrtimer_interrupt+0x128/0x2c0 [10804.138356] ? smp_apic_timer_interrupt+0xa6/0x270 [10804.140568] ? apic_timer_interrupt+0xf/0x20 [10804.142329] [10804.142954] ? native_safe_halt+0x2e/0x50 [10804.144061] kvm_wait+0xcd/0xe0 [10804.145508] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10804.147881] do_raw_spin_lock+0x177/0x190 [10804.149452] _raw_spin_lock+0x21/0x30 [10804.150638] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10804.152791] obd_mod_free+0x46/0x140 [ptlrpc] [10804.154765] mdc_commit_open+0x91/0xb0 [mdc] [10804.156118] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10804.157882] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10804.159892] after_reply+0xe2c/0x1780 [ptlrpc] [10804.161933] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10804.164790] ? __next_timer_interrupt+0x160/0x160 [10804.166664] ptlrpcd+0xa82/0xe40 [ptlrpc] [10804.168583] ? do_wait_intr+0xf0/0xf0 [10804.170253] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10804.172413] kthread+0x1d1/0x200 [10804.174784] ? set_kthread_struct+0x70/0x70 [10804.176080] ret_from_fork+0x1f/0x30 [10812.054869] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10812.063650] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10812.120316] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10812.133220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10812.150315] RIP: 0010:native_safe_halt+0x2e/0x50 [10812.154194] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10812.163546] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10812.173547] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10812.184410] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [10812.199819] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10812.210770] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10812.221582] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10812.231989] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10812.250690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10812.256361] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10812.259909] Call Trace: [10812.262340] [10812.263131] ? show_regs.cold.9+0x22/0x2f [10812.268608] ? watchdog_timer_fn.cold.6+0x62/0x10a [10812.271815] ? __hrtimer_run_queues+0x1c2/0x4d0 [10812.274670] ? watchdog+0x40/0x40 [10812.277651] ? hrtimer_interrupt+0x128/0x2c0 [10812.280946] ? smp_apic_timer_interrupt+0xa6/0x270 [10812.282658] ? apic_timer_interrupt+0xf/0x20 [10812.292971] [10812.297012] ? native_safe_halt+0x2e/0x50 [10812.304409] kvm_wait+0xcd/0xe0 [10812.308285] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10812.313488] do_raw_spin_lock+0x177/0x190 [10812.318962] _raw_spin_lock+0x21/0x30 [10812.322637] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10812.326717] ? do_raw_spin_unlock+0x75/0x190 [10812.330441] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10812.337652] ? do_wait_intr+0xf0/0xf0 [10812.340434] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10812.345761] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10812.356630] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10812.367486] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10812.373146] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10812.377446] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10812.385287] mdc_intent_lock+0x353/0x7e0 [mdc] [10812.387074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10812.390695] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10812.395653] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10812.401414] lmv_intent_open+0x338/0x1150 [lmv] [10812.406719] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10812.414692] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10812.423853] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10812.428491] ll_intent_lock+0xc5/0xaa0 [lustre] [10812.431440] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10812.435561] ll_lookup_it+0xe0e/0x2270 [lustre] [10812.439513] ? do_raw_spin_unlock+0x75/0x190 [10812.445416] ? _raw_spin_unlock+0x12/0x30 [10812.449520] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10812.451372] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10812.452707] ll_atomic_open+0x36a/0xd70 [lustre] [10812.456051] lookup_open+0x356/0xb40 [10812.458198] do_last+0x313/0xfc0 [10812.463847] ? nd_jump_root+0xe5/0x160 [10812.465302] ? path_init+0x437/0x520 [10812.466688] path_openat+0xf7/0x500 [10812.471407] ? do_raw_spin_unlock+0x75/0x190 [10812.473618] do_filp_open+0x99/0x140 [10812.475857] ? getname_flags+0x6e/0x330 [10812.478425] ? __check_object_size+0xff/0x256 [10812.481032] ? do_raw_spin_unlock+0x75/0x190 [10812.483318] ? _raw_spin_unlock+0x12/0x30 [10812.485411] do_sys_openat2+0x2b4/0x410 [10812.487559] do_sys_open+0x73/0xa0 [10812.489554] __x64_sys_openat+0x24/0x30 [10812.491612] do_syscall_64+0xc1/0x440 [10812.494570] entry_SYSCALL_64_after_hwframe+0x49/0xae [10812.497365] RIP: 0033:0x7f1a63983332 [10812.502655] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10812.513620] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10812.518157] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10812.521598] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10812.525356] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10812.528699] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10812.532469] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10817.506482] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8794s! [10817.524354] Showing busy workqueues and worker pools: [10817.530965] workqueue events: flags=0x0 [10817.532516] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10817.540876] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10817.558388] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10817.565601] pending: sec_gc_main [ptlrpc] [10817.567451] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10817.585727] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10817.595202] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10817.607186] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10817.617914] workqueue events_long: flags=0x0 [10817.620121] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10817.625370] pending: lu_objects_flush [obdclass] [10817.634355] workqueue events_unbound: flags=0x2 [10817.637755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10817.643227] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10817.657237] workqueue events_power_efficient: flags=0x80 [10817.663301] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10817.667373] pending: neigh_periodic_work, do_cache_clean [10817.670930] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10817.674175] pending: neigh_periodic_work, check_lifetime [10817.679747] workqueue mm_percpu_wq: flags=0x8 [10817.687876] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10817.695361] pending: vmstat_update [10817.699112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10817.701571] pending: vmstat_update [10817.708155] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10817.716150] pending: vmstat_update [10817.718491] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10817.725439] pending: vmstat_update [10817.726714] workqueue writeback: flags=0x4a [10817.730295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10817.733970] pending: wb_workfn, wb_workfn [10817.737305] workqueue xprtiod: flags=0xa [10817.740956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10817.744947] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10817.749650] workqueue ipv6_addrconf: flags=0x40008 [10817.752353] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10817.756618] pending: addrconf_verify_work [10817.759709] workqueue knbd0-recv: flags=0x1a [10817.761979] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10817.764706] in-flight: 647:recv_work [10817.768373] workqueue ptlrpc_pinger: flags=0x2 [10817.773794] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10817.780372] pending: ptlrpc_pinger_main [ptlrpc] [10817.784673] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10807s workers=2 idle: 652 [10820.056060] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10820.058719] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10820.059839] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10820.059912] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10820.059918] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10820.059921] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10820.059934] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10820.059939] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10820.059943] RAX: 0000000000007837 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10820.059946] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10820.059948] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10820.059951] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10820.059954] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10820.059957] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10820.059960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10820.059963] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10820.059971] Call Trace: [10820.059977] [10820.059994] ? show_regs.cold.9+0x22/0x2f [10820.060002] ? watchdog_timer_fn.cold.6+0x62/0x10a [10820.060009] ? __hrtimer_run_queues+0x1c2/0x4d0 [10820.060012] ? watchdog+0x40/0x40 [10820.060017] ? hrtimer_interrupt+0x128/0x2c0 [10820.060022] ? smp_apic_timer_interrupt+0xa6/0x270 [10820.060031] ? apic_timer_interrupt+0xf/0x20 [10820.060036] [10820.060162] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10820.060171] do_raw_spin_lock+0x177/0x190 [10820.060176] _raw_spin_lock+0x21/0x30 [10820.060181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10820.060599] ? do_raw_spin_unlock+0x75/0x190 [10820.060606] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10820.065456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10820.065535] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10820.065541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10820.065544] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10820.065558] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10820.065562] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10820.065565] RAX: 0000000000006b27 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10820.065568] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10820.065570] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10820.065572] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10820.065575] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10820.065578] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10820.065581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10820.065583] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10820.066106] Call Trace: [10820.066113] [10820.066117] ? show_regs.cold.9+0x22/0x2f [10820.066125] ? watchdog_timer_fn.cold.6+0x62/0x10a [10820.066129] ? __hrtimer_run_queues+0x1c2/0x4d0 [10820.066132] ? watchdog+0x40/0x40 [10820.066135] ? hrtimer_interrupt+0x128/0x2c0 [10820.066138] ? smp_apic_timer_interrupt+0xa6/0x270 [10820.066144] ? apic_timer_interrupt+0xf/0x20 [10820.066147] [10820.066149] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10820.066154] do_raw_spin_lock+0x177/0x190 [10820.066157] _raw_spin_lock+0x21/0x30 [10820.066160] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10820.066422] ? do_raw_spin_unlock+0x75/0x190 [10820.066427] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10820.068137] ? do_wait_intr+0xf0/0xf0 [10820.068145] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10820.068372] mdc_close+0x53b/0xd70 [mdc] [10820.068416] lmv_close+0x18d/0x490 [lmv] [10820.068441] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10820.068551] ll_md_real_close+0xa5/0x2a0 [lustre] [10820.070196] ll_md_close+0x274/0x800 [lustre] [10820.070279] ll_file_release+0x128/0xc50 [lustre] [10820.070362] __fput+0x10d/0x440 [10820.070371] ____fput+0x12/0x20 [10820.070376] task_work_run+0xc6/0x110 [10820.070383] exit_to_usermode_loop+0x1dd/0x1f0 [10820.070389] do_syscall_64+0x3d6/0x440 [10820.070393] entry_SYSCALL_64_after_hwframe+0x49/0xae [10820.070402] RIP: 0033:0x7f1a63983cd8 [10820.070409] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10820.070412] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10820.070417] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10820.070420] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10820.070422] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10820.070424] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10820.070426] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10820.445481] ? do_wait_intr+0xf0/0xf0 [10820.446802] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10820.450860] mdc_close+0x53b/0xd70 [mdc] [10820.456415] lmv_close+0x18d/0x490 [lmv] [10820.458234] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10820.464777] ll_md_real_close+0xa5/0x2a0 [lustre] [10820.476687] ll_md_close+0x274/0x800 [lustre] [10820.481954] ll_file_release+0x128/0xc50 [lustre] [10820.486264] __fput+0x10d/0x440 [10820.488770] ____fput+0x12/0x20 [10820.492575] task_work_run+0xc6/0x110 [10820.503061] exit_to_usermode_loop+0x1dd/0x1f0 [10820.511737] do_syscall_64+0x3d6/0x440 [10820.528661] entry_SYSCALL_64_after_hwframe+0x49/0xae [10820.530445] RIP: 0033:0x7f1a63983cd8 [10820.532644] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10820.552375] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10820.565845] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10820.567719] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10820.574486] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10820.578435] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10820.586738] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10832.059912] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [10832.069174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10832.098693] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10832.112493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10832.117970] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10832.122984] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10832.137082] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10832.146031] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10832.149601] RDX: 00000000000078b1 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10832.155304] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10832.161766] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10832.169657] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10832.176463] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10832.181255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10832.188066] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10832.194530] Call Trace: [10832.195846] [10832.197068] ? show_regs.cold.9+0x22/0x2f [10832.202734] ? watchdog_timer_fn.cold.6+0x62/0x10a [10832.206345] ? __hrtimer_run_queues+0x1c2/0x4d0 [10832.210723] ? watchdog+0x40/0x40 [10832.211999] ? hrtimer_interrupt+0x128/0x2c0 [10832.213587] ? smp_apic_timer_interrupt+0xa6/0x270 [10832.219570] ? apic_timer_interrupt+0xf/0x20 [10832.224058] [10832.224925] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10832.226734] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10832.239215] do_raw_spin_lock+0x177/0x190 [10832.248658] _raw_spin_lock+0x21/0x30 [10832.250102] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10832.257162] obd_mod_free+0x46/0x140 [ptlrpc] [10832.260800] mdc_commit_open+0x91/0xb0 [mdc] [10832.266061] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10832.268689] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10832.272813] after_reply+0xe2c/0x1780 [ptlrpc] [10832.274803] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10832.283139] ? __next_timer_interrupt+0x160/0x160 [10832.288143] ptlrpcd+0xa82/0xe40 [ptlrpc] [10832.294948] ? do_wait_intr+0xf0/0xf0 [10832.297354] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10832.304144] kthread+0x1d1/0x200 [10832.305341] ? set_kthread_struct+0x70/0x70 [10832.307728] ret_from_fork+0x1f/0x30 [10844.061496] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [10844.071223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10844.104592] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10844.116529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10844.127510] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10844.137417] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10844.155478] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10844.159239] RAX: 0000000000005685 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10844.162548] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10844.166066] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10844.169286] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10844.176094] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10844.181050] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10844.186796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10844.189623] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10844.192731] Call Trace: [10844.194008] [10844.194957] ? show_regs.cold.9+0x22/0x2f [10844.198816] ? watchdog_timer_fn.cold.6+0x62/0x10a [10844.204962] ? __hrtimer_run_queues+0x1c2/0x4d0 [10844.207717] ? watchdog+0x40/0x40 [10844.212979] ? hrtimer_interrupt+0x128/0x2c0 [10844.216389] ? smp_apic_timer_interrupt+0xa6/0x270 [10844.221459] ? apic_timer_interrupt+0xf/0x20 [10844.229680] [10844.233057] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10844.239941] do_raw_spin_lock+0x177/0x190 [10844.241901] _raw_spin_lock+0x21/0x30 [10844.243338] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10844.245482] ? do_raw_spin_unlock+0x75/0x190 [10844.251362] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10844.260078] ? do_wait_intr+0xf0/0xf0 [10844.262656] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10844.268123] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10844.274822] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10844.282106] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10844.285032] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10844.292643] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10844.298379] mdc_intent_lock+0x353/0x7e0 [mdc] [10844.303119] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10844.306616] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10844.320329] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10844.323983] lmv_intent_open+0x338/0x1150 [lmv] [10844.325865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10844.331913] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10844.338763] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10844.342872] ll_intent_lock+0xc5/0xaa0 [lustre] [10844.349999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10844.356873] ll_lookup_it+0xe0e/0x2270 [lustre] [10844.361707] ? do_raw_spin_unlock+0x75/0x190 [10844.365067] ? _raw_spin_unlock+0x12/0x30 [10844.369296] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10844.373163] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10844.378266] ll_atomic_open+0x36a/0xd70 [lustre] [10844.382126] lookup_open+0x356/0xb40 [10844.385258] do_last+0x313/0xfc0 [10844.388157] ? nd_jump_root+0xe5/0x160 [10844.390287] ? path_init+0x437/0x520 [10844.394302] path_openat+0xf7/0x500 [10844.398816] ? do_raw_spin_unlock+0x75/0x190 [10844.402342] do_filp_open+0x99/0x140 [10844.406076] ? getname_flags+0x6e/0x330 [10844.408321] ? __check_object_size+0xff/0x256 [10844.410350] ? do_raw_spin_unlock+0x75/0x190 [10844.414553] ? _raw_spin_unlock+0x12/0x30 [10844.419272] do_sys_openat2+0x2b4/0x410 [10844.422131] do_sys_open+0x73/0xa0 [10844.427144] __x64_sys_openat+0x24/0x30 [10844.431075] do_syscall_64+0xc1/0x440 [10844.435145] entry_SYSCALL_64_after_hwframe+0x49/0xae [10844.441219] RIP: 0033:0x7f1a63983332 [10844.444269] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10844.458497] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10844.470107] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10844.473979] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10844.487339] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10844.492931] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10844.499097] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10848.057895] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10848.070769] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10848.081848] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [10848.155107] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10848.155122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10848.155126] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10848.155142] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10848.155147] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10848.155153] RAX: 0000000000007586 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10848.160762] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10848.160850] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10848.160857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10848.160860] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10848.161000] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10848.161005] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10848.161009] RAX: 000000000000620c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10848.161012] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10848.161014] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10848.161015] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10848.161018] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10848.161021] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10848.161024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10848.161026] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10848.161035] Call Trace: [10848.161041] [10848.161047] ? show_regs.cold.9+0x22/0x2f [10848.161057] ? watchdog_timer_fn.cold.6+0x62/0x10a [10848.161064] ? __hrtimer_run_queues+0x1c2/0x4d0 [10848.161068] ? watchdog+0x40/0x40 [10848.161073] ? hrtimer_interrupt+0x128/0x2c0 [10848.161077] ? smp_apic_timer_interrupt+0xa6/0x270 [10848.161087] ? apic_timer_interrupt+0xf/0x20 [10848.161092] [10848.161095] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10848.161101] do_raw_spin_lock+0x177/0x190 [10848.161106] _raw_spin_lock+0x21/0x30 [10848.161112] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10848.161554] ? do_raw_spin_unlock+0x75/0x190 [10848.161560] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10848.163380] ? do_wait_intr+0xf0/0xf0 [10848.163395] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10848.164435] mdc_close+0x53b/0xd70 [mdc] [10848.164479] lmv_close+0x18d/0x490 [lmv] [10848.164520] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10848.164652] ll_md_real_close+0xa5/0x2a0 [lustre] [10848.164843] ll_md_close+0x274/0x800 [lustre] [10848.183325] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10848.183337] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10848.183343] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10848.183346] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10848.183349] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10848.183352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10848.183354] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10848.183363] Call Trace: [10848.183370] [10848.183375] ? show_regs.cold.9+0x22/0x2f [10848.183394] ? watchdog_timer_fn.cold.6+0x62/0x10a [10848.183402] ? __hrtimer_run_queues+0x1c2/0x4d0 [10848.183406] ? watchdog+0x40/0x40 [10848.183410] ? hrtimer_interrupt+0x128/0x2c0 [10848.183414] ? smp_apic_timer_interrupt+0xa6/0x270 [10848.183423] ? apic_timer_interrupt+0xf/0x20 [10848.183429] [10848.183432] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10848.183440] do_raw_spin_lock+0x177/0x190 [10848.183446] _raw_spin_lock+0x21/0x30 [10848.183457] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10848.184834] ? do_raw_spin_unlock+0x75/0x190 [10848.184847] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10848.187196] ? do_wait_intr+0xf0/0xf0 [10848.187268] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10848.187546] mdc_close+0x53b/0xd70 [mdc] [10848.187592] lmv_close+0x18d/0x490 [lmv] [10848.187623] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10848.188350] ll_md_real_close+0xa5/0x2a0 [lustre] [10848.188445] ll_md_close+0x274/0x800 [lustre] [10848.188565] ll_file_release+0x128/0xc50 [lustre] [10848.188829] __fput+0x10d/0x440 [10848.188838] ____fput+0x12/0x20 [10848.188843] task_work_run+0xc6/0x110 [10848.188850] exit_to_usermode_loop+0x1dd/0x1f0 [10848.188856] do_syscall_64+0x3d6/0x440 [10848.188860] entry_SYSCALL_64_after_hwframe+0x49/0xae [10848.188868] RIP: 0033:0x7f1a63983cd8 [10848.188875] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10848.188879] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10848.188885] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10848.188887] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10848.188890] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10848.188892] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10848.188894] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10848.809914] ll_file_release+0x128/0xc50 [lustre] [10848.817979] __fput+0x10d/0x440 [10848.825754] ____fput+0x12/0x20 [10848.826958] task_work_run+0xc6/0x110 [10848.832968] exit_to_usermode_loop+0x1dd/0x1f0 [10848.838545] do_syscall_64+0x3d6/0x440 [10848.844083] entry_SYSCALL_64_after_hwframe+0x49/0xae [10848.850851] RIP: 0033:0x7f1a63983cd8 [10848.853569] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10848.874943] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10848.881534] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10848.890720] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10848.897859] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10848.903736] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10848.910473] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10848.920057] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8825s! [10848.923187] Showing busy workqueues and worker pools: [10848.925882] workqueue events: flags=0x0 [10848.930619] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10848.938248] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10848.943266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10848.949835] pending: sec_gc_main [ptlrpc] [10848.952443] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10848.959924] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10848.963960] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10848.966574] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10848.971011] workqueue events_long: flags=0x0 [10848.972634] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10848.975610] pending: lu_objects_flush [obdclass] [10848.978291] workqueue events_unbound: flags=0x2 [10848.979997] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10848.983307] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10848.988304] workqueue events_power_efficient: flags=0x80 [10848.990566] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10848.993440] pending: neigh_periodic_work, do_cache_clean [10848.996000] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10848.998919] pending: neigh_periodic_work, check_lifetime [10849.001446] workqueue mm_percpu_wq: flags=0x8 [10849.002926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10849.005046] pending: vmstat_update [10849.006262] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10849.008489] pending: vmstat_update [10849.009954] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10849.014328] pending: vmstat_update [10849.016679] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10849.020019] pending: vmstat_update [10849.023631] workqueue writeback: flags=0x4a [10849.025198] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10849.040038] pending: wb_workfn, wb_workfn [10849.041908] workqueue xprtiod: flags=0xa [10849.049155] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10849.055463] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10849.061574] workqueue ipv6_addrconf: flags=0x40008 [10849.067887] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10849.074600] pending: addrconf_verify_work [10849.080158] workqueue knbd0-recv: flags=0x1a [10849.083278] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10849.090519] in-flight: 647:recv_work [10849.094462] workqueue ptlrpc_pinger: flags=0x2 [10849.100202] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10849.102098] pending: ptlrpc_pinger_main [ptlrpc] [10849.110280] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10839s workers=2 idle: 652 [10860.059520] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [10860.063362] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10860.099713] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10860.110837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10860.117508] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10860.121903] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10860.134615] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10860.139781] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10860.148115] RDX: 00000000000031f6 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10860.166786] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10860.175779] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10860.181268] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10860.198805] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10860.214652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10860.222267] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10860.231783] Call Trace: [10860.238060] [10860.244390] ? show_regs.cold.9+0x22/0x2f [10860.252100] ? watchdog_timer_fn.cold.6+0x62/0x10a [10860.262358] ? __hrtimer_run_queues+0x1c2/0x4d0 [10860.270005] ? watchdog+0x40/0x40 [10860.277387] ? hrtimer_interrupt+0x128/0x2c0 [10860.293437] ? smp_apic_timer_interrupt+0xa6/0x270 [10860.302973] ? apic_timer_interrupt+0xf/0x20 [10860.310735] [10860.313505] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10860.322352] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10860.337077] do_raw_spin_lock+0x177/0x190 [10860.350005] _raw_spin_lock+0x21/0x30 [10860.352110] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10860.359338] obd_mod_free+0x46/0x140 [ptlrpc] [10860.371706] mdc_commit_open+0x91/0xb0 [mdc] [10860.378576] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10860.387579] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10860.393762] after_reply+0xe2c/0x1780 [ptlrpc] [10860.400499] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10860.409277] ? __next_timer_interrupt+0x160/0x160 [10860.418333] ptlrpcd+0xa82/0xe40 [ptlrpc] [10860.429278] ? do_wait_intr+0xf0/0xf0 [10860.437084] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10860.445175] kthread+0x1d1/0x200 [10860.453572] ? set_kthread_struct+0x70/0x70 [10860.461899] ret_from_fork+0x1f/0x30 [10876.055341] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [10876.056019] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [10876.060013] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [10876.060025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10876.060109] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10876.060115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10876.060118] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10876.060144] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10876.060149] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10876.060153] RAX: 0000000000006325 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10876.060156] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10876.060158] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10876.060160] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10876.060162] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10876.060165] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10876.060168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10876.060170] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10876.060179] Call Trace: [10876.060184] [10876.060190] ? show_regs.cold.9+0x22/0x2f [10876.060199] ? watchdog_timer_fn.cold.6+0x62/0x10a [10876.060205] ? __hrtimer_run_queues+0x1c2/0x4d0 [10876.060209] ? watchdog+0x40/0x40 [10876.060213] ? hrtimer_interrupt+0x128/0x2c0 [10876.060218] ? smp_apic_timer_interrupt+0xa6/0x270 [10876.060226] ? apic_timer_interrupt+0xf/0x20 [10876.060231] [10876.060233] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10876.060239] do_raw_spin_lock+0x177/0x190 [10876.060243] _raw_spin_lock+0x21/0x30 [10876.060248] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10876.060525] ? do_raw_spin_unlock+0x75/0x190 [10876.060531] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10876.060741] ? do_wait_intr+0xf0/0xf0 [10876.060750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10876.060956] mdc_close+0x53b/0xd70 [mdc] [10876.060991] lmv_close+0x18d/0x490 [lmv] [10876.061024] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10876.061141] ll_md_real_close+0xa5/0x2a0 [lustre] [10876.061227] ll_md_close+0x274/0x800 [lustre] [10876.061312] ll_file_release+0x128/0xc50 [lustre] [10876.061393] __fput+0x10d/0x440 [10876.061401] ____fput+0x12/0x20 [10876.061405] task_work_run+0xc6/0x110 [10876.061412] exit_to_usermode_loop+0x1dd/0x1f0 [10876.061418] do_syscall_64+0x3d6/0x440 [10876.061422] entry_SYSCALL_64_after_hwframe+0x49/0xae [10876.061428] RIP: 0033:0x7f1a63983cd8 [10876.061447] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10876.061452] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10876.061455] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10876.061457] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10876.061459] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10876.061461] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10876.061463] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10876.068157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [10876.070933] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10876.071024] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10876.071035] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10876.071038] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10876.071056] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10876.071061] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10876.071066] RAX: 0000000000005590 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10876.071071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10876.071073] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10876.071075] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10876.071078] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10876.071084] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10876.071087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10876.071089] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10876.071097] Call Trace: [10876.071107] [10876.071113] ? show_regs.cold.9+0x22/0x2f [10876.071127] ? watchdog_timer_fn.cold.6+0x62/0x10a [10876.071133] ? __hrtimer_run_queues+0x1c2/0x4d0 [10876.071137] ? watchdog+0x40/0x40 [10876.071145] ? hrtimer_interrupt+0x128/0x2c0 [10876.071249] ? smp_apic_timer_interrupt+0xa6/0x270 [10876.071260] ? apic_timer_interrupt+0xf/0x20 [10876.071265] [10876.071268] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10876.071274] do_raw_spin_lock+0x177/0x190 [10876.071279] _raw_spin_lock+0x21/0x30 [10876.071288] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10876.073035] ? do_raw_spin_unlock+0x75/0x190 [10876.073043] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10876.073267] ? do_wait_intr+0xf0/0xf0 [10876.073275] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10876.074221] mdc_close+0x53b/0xd70 [mdc] [10876.074271] lmv_close+0x18d/0x490 [lmv] [10876.074305] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10876.074569] ll_md_real_close+0xa5/0x2a0 [lustre] [10876.074653] ll_md_close+0x274/0x800 [lustre] [10876.074737] ll_file_release+0x128/0xc50 [lustre] [10876.075188] __fput+0x10d/0x440 [10876.075198] ____fput+0x12/0x20 [10876.075203] task_work_run+0xc6/0x110 [10876.075211] exit_to_usermode_loop+0x1dd/0x1f0 [10876.075218] do_syscall_64+0x3d6/0x440 [10876.075226] entry_SYSCALL_64_after_hwframe+0x49/0xae [10876.075234] RIP: 0033:0x7f1a63983cd8 [10876.075242] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10876.075245] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10876.075250] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10876.075253] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10876.075255] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10876.075257] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10876.075259] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10876.668916] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10876.712862] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10876.719051] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10876.734652] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10876.740541] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10876.764159] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10876.770057] RAX: 0000000000006804 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10876.778090] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10876.789617] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10876.796902] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10876.801690] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10876.808224] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10876.812910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10876.816903] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10876.829371] Call Trace: [10876.833647] [10876.836782] ? show_regs.cold.9+0x22/0x2f [10876.844519] ? watchdog_timer_fn.cold.6+0x62/0x10a [10876.848243] ? __hrtimer_run_queues+0x1c2/0x4d0 [10876.850784] ? watchdog+0x40/0x40 [10876.852281] ? hrtimer_interrupt+0x128/0x2c0 [10876.854129] ? smp_apic_timer_interrupt+0xa6/0x270 [10876.864564] ? apic_timer_interrupt+0xf/0x20 [10876.869820] [10876.871055] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10876.876362] do_raw_spin_lock+0x177/0x190 [10876.878248] _raw_spin_lock+0x21/0x30 [10876.883258] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10876.891899] ? do_raw_spin_unlock+0x75/0x190 [10876.895806] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10876.904085] ? do_wait_intr+0xf0/0xf0 [10876.907506] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10876.913395] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10876.918109] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10876.928260] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10876.943376] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10876.952349] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10876.958324] mdc_intent_lock+0x353/0x7e0 [mdc] [10876.959953] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10876.975625] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10876.982020] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10876.990380] lmv_intent_open+0x338/0x1150 [lmv] [10876.996399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10877.009682] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10877.018313] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10877.020479] ll_intent_lock+0xc5/0xaa0 [lustre] [10877.027268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10877.031410] ll_lookup_it+0xe0e/0x2270 [lustre] [10877.046002] ? do_raw_spin_unlock+0x75/0x190 [10877.049674] ? _raw_spin_unlock+0x12/0x30 [10877.051397] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10877.055132] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10877.059024] ll_atomic_open+0x36a/0xd70 [lustre] [10877.061388] lookup_open+0x356/0xb40 [10877.063654] do_last+0x313/0xfc0 [10877.065806] ? nd_jump_root+0xe5/0x160 [10877.067245] ? path_init+0x437/0x520 [10877.075688] path_openat+0xf7/0x500 [10877.078401] ? do_raw_spin_unlock+0x75/0x190 [10877.082737] do_filp_open+0x99/0x140 [10877.086107] ? getname_flags+0x6e/0x330 [10877.088133] ? __check_object_size+0xff/0x256 [10877.091193] ? do_raw_spin_unlock+0x75/0x190 [10877.093422] ? _raw_spin_unlock+0x12/0x30 [10877.102670] do_sys_openat2+0x2b4/0x410 [10877.104061] do_sys_open+0x73/0xa0 [10877.105301] __x64_sys_openat+0x24/0x30 [10877.114081] do_syscall_64+0xc1/0x440 [10877.117037] entry_SYSCALL_64_after_hwframe+0x49/0xae [10877.119976] RIP: 0033:0x7f1a63983332 [10877.122105] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10877.138000] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10877.152956] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10877.174728] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10877.183376] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10877.195449] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10877.202755] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10879.460575] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8856s! [10879.465438] Showing busy workqueues and worker pools: [10879.469343] workqueue events: flags=0x0 [10879.471890] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10879.477245] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10879.483389] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10879.487463] pending: sec_gc_main [ptlrpc] [10879.493707] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10879.499826] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10879.513388] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10879.524541] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10879.543303] workqueue events_long: flags=0x0 [10879.547239] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10879.553726] pending: lu_objects_flush [obdclass] [10879.558772] workqueue events_unbound: flags=0x2 [10879.561728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10879.569817] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10879.581262] workqueue events_power_efficient: flags=0x80 [10879.588758] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10879.596139] pending: neigh_periodic_work, do_cache_clean [10879.601063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10879.604932] pending: neigh_periodic_work, check_lifetime [10879.610085] workqueue mm_percpu_wq: flags=0x8 [10879.613263] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10879.619239] pending: vmstat_update [10879.622213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10879.629178] pending: vmstat_update [10879.631920] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10879.635826] pending: vmstat_update [10879.638776] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10879.642472] pending: vmstat_update [10879.644076] workqueue writeback: flags=0x4a [10879.646455] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10879.650198] pending: wb_workfn, wb_workfn [10879.652570] workqueue xprtiod: flags=0xa [10879.654687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10879.658208] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10879.662213] workqueue ipv6_addrconf: flags=0x40008 [10879.665354] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10879.668313] pending: addrconf_verify_work [10879.671878] workqueue knbd0-recv: flags=0x1a [10879.674963] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10879.680184] in-flight: 647:recv_work [10879.683699] workqueue ptlrpc_pinger: flags=0x2 [10879.686060] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10879.689235] pending: ptlrpc_pinger_main [ptlrpc] [10879.693477] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10869s workers=2 idle: 652 [10888.058066] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [10888.062157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10888.106944] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10888.115906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10888.121669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10888.127108] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10888.145422] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10888.152736] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10888.160676] RDX: 000000000000703a RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10888.169875] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10888.175669] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10888.179489] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10888.189742] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10888.197638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10888.200305] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10888.205247] Call Trace: [10888.207808] [10888.209221] ? show_regs.cold.9+0x22/0x2f [10888.210708] ? watchdog_timer_fn.cold.6+0x62/0x10a [10888.218737] ? __hrtimer_run_queues+0x1c2/0x4d0 [10888.221483] ? watchdog+0x40/0x40 [10888.222657] ? hrtimer_interrupt+0x128/0x2c0 [10888.223975] ? smp_apic_timer_interrupt+0xa6/0x270 [10888.226680] ? apic_timer_interrupt+0xf/0x20 [10888.229828] [10888.231475] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10888.234200] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10888.244954] do_raw_spin_lock+0x177/0x190 [10888.249651] _raw_spin_lock+0x21/0x30 [10888.253263] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10888.259726] obd_mod_free+0x46/0x140 [ptlrpc] [10888.261615] mdc_commit_open+0x91/0xb0 [mdc] [10888.263154] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10888.269609] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10888.274752] after_reply+0xe2c/0x1780 [ptlrpc] [10888.278219] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10888.287835] ? __next_timer_interrupt+0x160/0x160 [10888.290767] ptlrpcd+0xa82/0xe40 [ptlrpc] [10888.293648] ? do_wait_intr+0xf0/0xf0 [10888.295470] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10888.297981] kthread+0x1d1/0x200 [10888.299650] ? set_kthread_struct+0x70/0x70 [10888.301744] ret_from_fork+0x1f/0x30 [10902.648166] rcu: INFO: rcu_sched self-detected stall on CPU [10902.650733] rcu: 3-...!: (5603160 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [10902.663087] (t=8880255 jiffies g=269633 q=3195) [10902.668069] rcu: rcu_sched kthread starved for 8879905 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [10902.676400] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10902.680016] rcu: RCU grace-period kthread stack dump: [10902.682326] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10902.686403] Call Trace: [10902.687348] __schedule+0x351/0xcb0 [10902.688741] schedule+0xc0/0x180 [10902.690004] schedule_timeout+0xb4/0x190 [10902.691718] ? do_raw_spin_unlock+0x75/0x190 [10902.693303] ? __next_timer_interrupt+0x160/0x160 [10902.695385] ? prepare_to_swait_event+0xf3/0x140 [10902.697285] rcu_gp_kthread+0x8dd/0x15b0 [10902.698903] ? force_qs_rnp+0x2c0/0x2c0 [10902.700660] kthread+0x1d1/0x200 [10902.702050] ? set_kthread_struct+0x70/0x70 [10902.703935] ret_from_fork+0x1f/0x30 [10902.705661] rcu: Stack dump where RCU GP kthread last ran: [10902.707970] Sending NMI from CPU 3 to CPUs 2: [10902.710799] NMI backtrace for cpu 2 [10902.710804] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10902.710807] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10902.710808] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10902.710810] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10902.710813] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 [10902.710815] RAX: 0000000000004830 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10902.710816] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10902.710818] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10902.710819] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10902.710820] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10902.710821] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10902.710822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10902.710824] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10902.710825] Call Trace: [10902.710826] [10902.710826] ? show_regs.cold.9+0x22/0x2f [10902.710827] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10902.710828] ? nmi_cpu_backtrace_handler+0x15/0x40 [10902.710829] ? nmi_handle+0xbd/0x210 [10902.710830] ? default_do_nmi+0x4b/0x1b0 [10902.710831] ? do_nmi+0x280/0x380 [10902.710832] ? end_repeat_nmi+0x16/0x1f [10902.710833] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10902.710834] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10902.710835] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10902.710836] [10902.710836] do_raw_spin_lock+0x177/0x190 [10902.710837] _raw_spin_lock+0x21/0x30 [10902.710838] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10902.710839] ? do_raw_spin_unlock+0x75/0x190 [10902.710840] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10902.710841] ? do_wait_intr+0xf0/0xf0 [10902.710842] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10902.710843] mdc_close+0x53b/0xd70 [mdc] [10902.710844] lmv_close+0x18d/0x490 [lmv] [10902.710845] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10902.710845] ll_md_real_close+0xa5/0x2a0 [lustre] [10902.710846] ll_md_close+0x274/0x800 [lustre] [10902.710847] ll_file_release+0x128/0xc50 [lustre] [10902.710848] __fput+0x10d/0x440 [10902.710849] ____fput+0x12/0x20 [10902.710850] task_work_run+0xc6/0x110 [10902.710851] exit_to_usermode_loop+0x1dd/0x1f0 [10902.710851] do_syscall_64+0x3d6/0x440 [10902.710852] entry_SYSCALL_64_after_hwframe+0x49/0xae [10902.710853] RIP: 0033:0x7f1a63983cd8 [10902.710854] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10902.711055] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10902.711058] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10902.711059] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10902.711060] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10902.711061] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10902.711063] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10902.711287] NMI backtrace for cpu 3 [10902.946765] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10902.956554] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10902.962203] Call Trace: [10902.964105] [10902.965579] ? dump_stack+0xbb/0x10e [10902.972674] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10902.980336] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10902.988345] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10902.996704] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10903.000071] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10903.004816] ? rcu_dump_cpu_stacks+0x11b/0x171 [10903.008315] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10903.013732] ? update_process_times+0x6d/0xc0 [10903.022872] ? tick_sched_handle.isra.18+0x27/0x90 [10903.035884] ? tick_sched_timer+0x53/0xb0 [10903.037432] ? __hrtimer_run_queues+0x1c2/0x4d0 [10903.049776] ? tick_sched_handle.isra.18+0x90/0x90 [10903.060786] ? hrtimer_interrupt+0x128/0x2c0 [10903.064423] ? smp_apic_timer_interrupt+0xa6/0x270 [10903.069073] ? apic_timer_interrupt+0xf/0x20 [10903.070770] [10903.071634] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10903.083065] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10903.084992] ? do_raw_spin_lock+0x177/0x190 [10903.096072] ? _raw_spin_lock+0x21/0x30 [10903.098403] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10903.105827] ? obd_mod_free+0x46/0x140 [ptlrpc] [10903.119859] ? mdc_commit_open+0x91/0xb0 [mdc] [10903.129364] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10903.132707] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10903.146633] ? after_reply+0xe2c/0x1780 [ptlrpc] [10903.154239] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10903.162804] ? __next_timer_interrupt+0x160/0x160 [10903.167105] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10903.174249] ? do_wait_intr+0xf0/0xf0 [10903.180453] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10903.193147] ? kthread+0x1d1/0x200 [10903.196109] ? set_kthread_struct+0x70/0x70 [10903.201402] ? ret_from_fork+0x1f/0x30 [10904.053381] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [10904.058535] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [10904.066340] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10904.066426] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10904.066438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10904.066441] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10904.066456] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10904.066460] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10904.066465] RAX: 0000000000006583 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10904.066468] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10904.066470] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10904.066473] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10904.066475] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10904.066479] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10904.066482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10904.066485] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10904.066493] Call Trace: [10904.066498] [10904.066504] ? show_regs.cold.9+0x22/0x2f [10904.066513] ? watchdog_timer_fn.cold.6+0x62/0x10a [10904.066520] ? __hrtimer_run_queues+0x1c2/0x4d0 [10904.066524] ? watchdog+0x40/0x40 [10904.066528] ? hrtimer_interrupt+0x128/0x2c0 [10904.066545] ? smp_apic_timer_interrupt+0xa6/0x270 [10904.066554] ? apic_timer_interrupt+0xf/0x20 [10904.066559] [10904.066562] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10904.066569] do_raw_spin_lock+0x177/0x190 [10904.066573] _raw_spin_lock+0x21/0x30 [10904.066578] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10904.066842] ? do_raw_spin_unlock+0x75/0x190 [10904.066848] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10904.067443] ? do_wait_intr+0xf0/0xf0 [10904.067452] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10904.067966] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10904.068863] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10904.076098] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [10904.110413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.110542] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10904.110612] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10904.110644] mdc_intent_lock+0x353/0x7e0 [mdc] [10904.110674] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.110782] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10904.111120] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10904.111172] lmv_intent_open+0x338/0x1150 [lmv] [10904.111207] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.111376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.111542] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10904.111566] ll_intent_lock+0xc5/0xaa0 [lustre] [10904.111675] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.111772] ll_lookup_it+0xe0e/0x2270 [lustre] [10904.111867] ? do_raw_spin_unlock+0x75/0x190 [10904.111875] ? _raw_spin_unlock+0x12/0x30 [10904.111881] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10904.111910] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10904.111929] ll_atomic_open+0x36a/0xd70 [lustre] [10904.112024] lookup_open+0x356/0xb40 [10904.112034] do_last+0x313/0xfc0 [10904.112039] ? nd_jump_root+0xe5/0x160 [10904.112043] ? path_init+0x437/0x520 [10904.112048] path_openat+0xf7/0x500 [10904.112053] ? do_raw_spin_unlock+0x75/0x190 [10904.112058] do_filp_open+0x99/0x140 [10904.112064] ? getname_flags+0x6e/0x330 [10904.112068] ? __check_object_size+0xff/0x256 [10904.112414] ? do_raw_spin_unlock+0x75/0x190 [10904.112419] ? _raw_spin_unlock+0x12/0x30 [10904.112424] do_sys_openat2+0x2b4/0x410 [10904.112429] do_sys_open+0x73/0xa0 [10904.112432] __x64_sys_openat+0x24/0x30 [10904.112436] do_syscall_64+0xc1/0x440 [10904.112442] entry_SYSCALL_64_after_hwframe+0x49/0xae [10904.112448] RIP: 0033:0x7f1a63983332 [10904.112455] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10904.112459] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10904.112464] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10904.112467] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10904.112469] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10904.112471] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10904.112473] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10904.402597] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10904.425673] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10904.435174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10904.442150] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10904.446795] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10904.461898] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10904.466102] RAX: 000000000000368a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10904.469992] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10904.474184] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10904.477911] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10904.480562] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10904.485358] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10904.488802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10904.491437] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10904.493825] Call Trace: [10904.494834] [10904.495632] ? show_regs.cold.9+0x22/0x2f [10904.497121] ? watchdog_timer_fn.cold.6+0x62/0x10a [10904.498999] ? __hrtimer_run_queues+0x1c2/0x4d0 [10904.501442] ? watchdog+0x40/0x40 [10904.504488] ? hrtimer_interrupt+0x128/0x2c0 [10904.506969] ? smp_apic_timer_interrupt+0xa6/0x270 [10904.511253] ? apic_timer_interrupt+0xf/0x20 [10904.513821] [10904.514833] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10904.519815] do_raw_spin_lock+0x177/0x190 [10904.521417] _raw_spin_lock+0x21/0x30 [10904.523555] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10904.528836] ? do_raw_spin_unlock+0x75/0x190 [10904.533556] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10904.536984] ? do_wait_intr+0xf0/0xf0 [10904.539566] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10904.542895] mdc_close+0x53b/0xd70 [mdc] [10904.547564] lmv_close+0x18d/0x490 [lmv] [10904.552098] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10904.556903] ll_md_real_close+0xa5/0x2a0 [lustre] [10904.561791] ll_md_close+0x274/0x800 [lustre] [10904.565436] ll_file_release+0x128/0xc50 [lustre] [10904.568023] __fput+0x10d/0x440 [10904.569482] ____fput+0x12/0x20 [10904.571860] task_work_run+0xc6/0x110 [10904.573698] exit_to_usermode_loop+0x1dd/0x1f0 [10904.575944] do_syscall_64+0x3d6/0x440 [10904.577568] entry_SYSCALL_64_after_hwframe+0x49/0xae [10904.581896] RIP: 0033:0x7f1a63983cd8 [10904.583995] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10904.597809] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10904.605943] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10904.609794] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10904.620179] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10904.624989] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10904.628279] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10910.178657] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8887s! [10910.188269] Showing busy workqueues and worker pools: [10910.192344] workqueue events: flags=0x0 [10910.195079] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10910.203665] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10910.214634] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10910.227120] pending: sec_gc_main [ptlrpc] [10910.233029] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10910.239328] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10910.259017] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10910.270776] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10910.282351] workqueue events_long: flags=0x0 [10910.284468] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10910.297741] pending: lu_objects_flush [obdclass] [10910.301618] workqueue events_unbound: flags=0x2 [10910.304570] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10910.308384] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10910.319719] workqueue events_power_efficient: flags=0x80 [10910.334495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10910.336966] pending: neigh_periodic_work, do_cache_clean [10910.345807] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10910.354807] pending: neigh_periodic_work, check_lifetime [10910.361604] workqueue mm_percpu_wq: flags=0x8 [10910.366745] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10910.377782] pending: vmstat_update [10910.382245] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10910.386323] pending: vmstat_update [10910.388835] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10910.397360] pending: vmstat_update [10910.403658] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10910.411051] pending: vmstat_update [10910.413687] workqueue writeback: flags=0x4a [10910.419053] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10910.423461] pending: wb_workfn, wb_workfn [10910.429664] workqueue xprtiod: flags=0xa [10910.433934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10910.442285] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10910.455270] workqueue ipv6_addrconf: flags=0x40008 [10910.462795] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10910.470808] pending: addrconf_verify_work [10910.478178] workqueue knbd0-recv: flags=0x1a [10910.480590] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10910.489307] in-flight: 647:recv_work [10910.496317] workqueue ptlrpc_pinger: flags=0x2 [10910.500913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10910.510838] pending: ptlrpc_pinger_main [ptlrpc] [10910.516278] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10900s workers=2 idle: 652 [10928.058091] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10928.060402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10928.072172] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10928.112402] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10928.112416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10928.112419] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10928.112433] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10928.115763] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10928.115861] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10928.115868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10928.115871] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10928.115886] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10928.115891] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10928.115895] RAX: 0000000000007e9e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10928.115898] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10928.115900] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10928.115902] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10928.115905] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10928.115908] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10928.115911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10928.115913] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10928.115939] Call Trace: [10928.115946] [10928.115952] ? show_regs.cold.9+0x22/0x2f [10928.115963] ? watchdog_timer_fn.cold.6+0x62/0x10a [10928.115976] ? __hrtimer_run_queues+0x1c2/0x4d0 [10928.115980] ? watchdog+0x40/0x40 [10928.115985] ? hrtimer_interrupt+0x128/0x2c0 [10928.115989] ? smp_apic_timer_interrupt+0xa6/0x270 [10928.115998] ? apic_timer_interrupt+0xf/0x20 [10928.116003] [10928.116006] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10928.116012] do_raw_spin_lock+0x177/0x190 [10928.116018] _raw_spin_lock+0x21/0x30 [10928.116025] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10928.116347] ? do_raw_spin_unlock+0x75/0x190 [10928.116357] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10928.116577] ? do_wait_intr+0xf0/0xf0 [10928.116586] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10928.116819] mdc_close+0x53b/0xd70 [mdc] [10928.116858] lmv_close+0x18d/0x490 [lmv] [10928.116898] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10928.117043] ll_md_real_close+0xa5/0x2a0 [lustre] [10928.117222] ll_md_close+0x274/0x800 [lustre] [10928.117322] ll_file_release+0x128/0xc50 [lustre] [10928.117418] __fput+0x10d/0x440 [10928.117433] ____fput+0x12/0x20 [10928.117437] task_work_run+0xc6/0x110 [10928.117445] exit_to_usermode_loop+0x1dd/0x1f0 [10928.117451] do_syscall_64+0x3d6/0x440 [10928.117456] entry_SYSCALL_64_after_hwframe+0x49/0xae [10928.117462] RIP: 0033:0x7f1a63983cd8 [10928.117468] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10928.117472] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10928.117477] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10928.117480] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10928.117482] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10928.117484] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10928.117486] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10928.317189] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10928.321369] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10928.325273] RDX: 00000000000055a0 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10928.329553] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10928.333221] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10928.337123] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10928.340448] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10928.344451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10928.347525] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10928.350897] Call Trace: [10928.351894] [10928.353318] ? show_regs.cold.9+0x22/0x2f [10928.355449] ? watchdog_timer_fn.cold.6+0x62/0x10a [10928.358083] ? __hrtimer_run_queues+0x1c2/0x4d0 [10928.360553] ? watchdog+0x40/0x40 [10928.362490] ? hrtimer_interrupt+0x128/0x2c0 [10928.365383] ? smp_apic_timer_interrupt+0xa6/0x270 [10928.367799] ? apic_timer_interrupt+0xf/0x20 [10928.370065] [10928.371038] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10928.373806] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10928.377158] do_raw_spin_lock+0x177/0x190 [10928.379116] _raw_spin_lock+0x21/0x30 [10928.381534] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10928.384458] obd_mod_free+0x46/0x140 [ptlrpc] [10928.387006] mdc_commit_open+0x91/0xb0 [mdc] [10928.389598] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10928.392361] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10928.395273] after_reply+0xe2c/0x1780 [ptlrpc] [10928.398040] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10928.401028] ? __next_timer_interrupt+0x160/0x160 [10928.403449] ptlrpcd+0xa82/0xe40 [ptlrpc] [10928.406154] ? do_wait_intr+0xf0/0xf0 [10928.407837] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10928.410563] kthread+0x1d1/0x200 [10928.412087] ? set_kthread_struct+0x70/0x70 [10928.415180] ret_from_fork+0x1f/0x30 [10932.057163] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [10932.064038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10932.092682] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10932.097531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10932.101423] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10932.104455] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10932.114628] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10932.118303] RAX: 0000000000000507 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10932.121563] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10932.127773] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10932.134675] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10932.138567] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10932.143932] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10932.148561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10932.151776] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10932.155650] Call Trace: [10932.157069] [10932.157903] ? show_regs.cold.9+0x22/0x2f [10932.159806] ? watchdog_timer_fn.cold.6+0x62/0x10a [10932.161591] ? __hrtimer_run_queues+0x1c2/0x4d0 [10932.163804] ? watchdog+0x40/0x40 [10932.165515] ? hrtimer_interrupt+0x128/0x2c0 [10932.167779] ? smp_apic_timer_interrupt+0xa6/0x270 [10932.170463] ? apic_timer_interrupt+0xf/0x20 [10932.172295] [10932.173024] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10932.175878] do_raw_spin_lock+0x177/0x190 [10932.177927] _raw_spin_lock+0x21/0x30 [10932.180001] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10932.183403] ? do_raw_spin_unlock+0x75/0x190 [10932.185225] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10932.187653] ? do_wait_intr+0xf0/0xf0 [10932.191434] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10932.194539] mdc_close+0x53b/0xd70 [mdc] [10932.196605] lmv_close+0x18d/0x490 [lmv] [10932.198680] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10932.201608] ll_md_real_close+0xa5/0x2a0 [lustre] [10932.204554] ll_md_close+0x274/0x800 [lustre] [10932.207259] ll_file_release+0x128/0xc50 [lustre] [10932.210210] __fput+0x10d/0x440 [10932.212129] ____fput+0x12/0x20 [10932.213689] task_work_run+0xc6/0x110 [10932.215666] exit_to_usermode_loop+0x1dd/0x1f0 [10932.218519] do_syscall_64+0x3d6/0x440 [10932.220649] entry_SYSCALL_64_after_hwframe+0x49/0xae [10932.223798] RIP: 0033:0x7f1a63983cd8 [10932.225684] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10932.246393] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10932.250730] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10932.261067] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10932.270691] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10932.277105] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10932.280733] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10936.054898] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10936.058411] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10936.087455] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10936.094688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10936.101393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10936.106637] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10936.122861] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10936.129546] RAX: 0000000000007812 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10936.132553] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10936.135982] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10936.139869] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10936.143145] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10936.146279] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10936.150039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10936.152501] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10936.155780] Call Trace: [10936.157056] [10936.158023] ? show_regs.cold.9+0x22/0x2f [10936.160399] ? watchdog_timer_fn.cold.6+0x62/0x10a [10936.162400] ? __hrtimer_run_queues+0x1c2/0x4d0 [10936.164639] ? watchdog+0x40/0x40 [10936.166348] ? hrtimer_interrupt+0x128/0x2c0 [10936.168829] ? smp_apic_timer_interrupt+0xa6/0x270 [10936.170840] ? apic_timer_interrupt+0xf/0x20 [10936.172713] [10936.173971] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10936.176262] do_raw_spin_lock+0x177/0x190 [10936.178119] _raw_spin_lock+0x21/0x30 [10936.179368] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10936.181912] ? do_raw_spin_unlock+0x75/0x190 [10936.184042] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10936.186491] ? do_wait_intr+0xf0/0xf0 [10936.188453] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10936.190801] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10936.193438] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10936.196253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10936.198368] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10936.200988] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10936.203065] mdc_intent_lock+0x353/0x7e0 [mdc] [10936.205394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10936.207816] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10936.211295] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10936.213648] lmv_intent_open+0x338/0x1150 [lmv] [10936.215912] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10936.218436] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10936.221016] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10936.222966] ll_intent_lock+0xc5/0xaa0 [lustre] [10936.225214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10936.227489] ll_lookup_it+0xe0e/0x2270 [lustre] [10936.229320] ? do_raw_spin_unlock+0x75/0x190 [10936.230601] ? _raw_spin_unlock+0x12/0x30 [10936.231724] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10936.234399] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10936.237200] ll_atomic_open+0x36a/0xd70 [lustre] [10936.241379] lookup_open+0x356/0xb40 [10936.243647] do_last+0x313/0xfc0 [10936.245676] ? nd_jump_root+0xe5/0x160 [10936.248262] ? path_init+0x437/0x520 [10936.249682] path_openat+0xf7/0x500 [10936.251483] ? do_raw_spin_unlock+0x75/0x190 [10936.253450] do_filp_open+0x99/0x140 [10936.255423] ? getname_flags+0x6e/0x330 [10936.257293] ? __check_object_size+0xff/0x256 [10936.259635] ? do_raw_spin_unlock+0x75/0x190 [10936.261937] ? _raw_spin_unlock+0x12/0x30 [10936.264008] do_sys_openat2+0x2b4/0x410 [10936.265929] do_sys_open+0x73/0xa0 [10936.267478] __x64_sys_openat+0x24/0x30 [10936.269435] do_syscall_64+0xc1/0x440 [10936.271328] entry_SYSCALL_64_after_hwframe+0x49/0xae [10936.274539] RIP: 0033:0x7f1a63983332 [10936.276328] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10936.289239] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10936.293354] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10936.297186] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10936.300617] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10936.304412] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10936.307977] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10940.900049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8917s! [10940.903018] Showing busy workqueues and worker pools: [10940.907160] workqueue events: flags=0x0 [10940.909354] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10940.914975] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10940.923386] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10940.936371] pending: sec_gc_main [ptlrpc] [10940.943665] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10940.950354] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10940.956288] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10940.961223] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10940.974372] workqueue events_long: flags=0x0 [10940.978360] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10940.986145] pending: lu_objects_flush [obdclass] [10940.994324] workqueue events_unbound: flags=0x2 [10940.995943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10941.008139] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10941.012693] workqueue events_power_efficient: flags=0x80 [10941.021225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10941.025117] pending: neigh_periodic_work, do_cache_clean [10941.030035] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10941.034287] pending: neigh_periodic_work, check_lifetime [10941.038508] workqueue mm_percpu_wq: flags=0x8 [10941.041046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10941.048245] pending: vmstat_update [10941.050251] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10941.054043] pending: vmstat_update [10941.055956] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10941.059215] pending: vmstat_update [10941.061323] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10941.065158] pending: vmstat_update [10941.067131] workqueue writeback: flags=0x4a [10941.069454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10941.072476] pending: wb_workfn, wb_workfn [10941.074590] workqueue xprtiod: flags=0xa [10941.077045] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10941.080092] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10941.083646] workqueue ipv6_addrconf: flags=0x40008 [10941.087053] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10941.089756] pending: addrconf_verify_work [10941.092417] workqueue knbd0-recv: flags=0x1a [10941.096917] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10941.104371] in-flight: 647:recv_work [10941.105755] workqueue ptlrpc_pinger: flags=0x2 [10941.117276] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10941.126106] pending: ptlrpc_pinger_main [ptlrpc] [10941.137140] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10931s workers=2 idle: 652 [10956.057147] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10956.058719] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10956.062475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10956.062735] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10956.062746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10956.062749] RIP: 0010:native_safe_halt+0x2e/0x50 [10956.062803] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10956.062808] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10956.062812] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10956.062818] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [10956.062820] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10956.062822] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10956.062825] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10956.062828] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10956.062831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10956.062837] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10956.062845] Call Trace: [10956.062863] [10956.062869] ? show_regs.cold.9+0x22/0x2f [10956.062887] ? watchdog_timer_fn.cold.6+0x62/0x10a [10956.062895] ? __hrtimer_run_queues+0x1c2/0x4d0 [10956.062899] ? watchdog+0x40/0x40 [10956.062907] ? hrtimer_interrupt+0x128/0x2c0 [10956.062911] ? smp_apic_timer_interrupt+0xa6/0x270 [10956.062922] ? apic_timer_interrupt+0xf/0x20 [10956.062926] [10956.062930] ? native_safe_halt+0x2e/0x50 [10956.062934] kvm_wait+0xcd/0xe0 [10956.062946] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10956.062954] do_raw_spin_lock+0x177/0x190 [10956.062961] _raw_spin_lock+0x21/0x30 [10956.062966] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10956.063462] ? do_raw_spin_unlock+0x75/0x190 [10956.063468] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10956.063754] ? do_wait_intr+0xf0/0xf0 [10956.063763] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10956.063993] mdc_close+0x53b/0xd70 [mdc] [10956.064045] lmv_close+0x18d/0x490 [lmv] [10956.064157] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10956.064311] ll_md_real_close+0xa5/0x2a0 [lustre] [10956.064399] ll_md_close+0x274/0x800 [lustre] [10956.064500] ll_file_release+0x128/0xc50 [lustre] [10956.064589] __fput+0x10d/0x440 [10956.064597] ____fput+0x12/0x20 [10956.064602] task_work_run+0xc6/0x110 [10956.064610] exit_to_usermode_loop+0x1dd/0x1f0 [10956.064618] do_syscall_64+0x3d6/0x440 [10956.064622] entry_SYSCALL_64_after_hwframe+0x49/0xae [10956.064631] RIP: 0033:0x7f1a63983cd8 [10956.064640] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10956.064644] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10956.064649] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10956.064718] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10956.064722] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10956.064724] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10956.064726] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10956.304861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10956.332863] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10956.338123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10956.342817] RIP: 0010:native_safe_halt+0x2e/0x50 [10956.345515] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10956.352606] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10956.356455] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10956.360458] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10956.364393] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10956.368691] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10956.372347] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10956.375943] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10956.380728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10956.383846] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10956.388047] Call Trace: [10956.389672] [10956.390665] ? show_regs.cold.9+0x22/0x2f [10956.392679] ? watchdog_timer_fn.cold.6+0x62/0x10a [10956.394928] ? __hrtimer_run_queues+0x1c2/0x4d0 [10956.397372] ? watchdog+0x40/0x40 [10956.399356] ? hrtimer_interrupt+0x128/0x2c0 [10956.401275] ? smp_apic_timer_interrupt+0xa6/0x270 [10956.403780] ? apic_timer_interrupt+0xf/0x20 [10956.406150] [10956.406972] ? native_safe_halt+0x2e/0x50 [10956.409233] kvm_wait+0xcd/0xe0 [10956.410779] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10956.413546] do_raw_spin_lock+0x177/0x190 [10956.415586] _raw_spin_lock+0x21/0x30 [10956.417311] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10956.419570] obd_mod_free+0x46/0x140 [ptlrpc] [10956.421375] mdc_commit_open+0x91/0xb0 [mdc] [10956.423612] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10956.426287] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10956.429275] after_reply+0xe2c/0x1780 [ptlrpc] [10956.431536] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10956.434782] ? __next_timer_interrupt+0x160/0x160 [10956.438314] ptlrpcd+0xa82/0xe40 [ptlrpc] [10956.440570] ? do_wait_intr+0xf0/0xf0 [10956.442121] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10956.444700] kthread+0x1d1/0x200 [10956.446248] ? set_kthread_struct+0x70/0x70 [10956.447958] ret_from_fork+0x1f/0x30 [10960.056018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [10960.060983] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10960.088124] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10960.097182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10960.101721] RIP: 0010:native_safe_halt+0x2e/0x50 [10960.103504] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10960.116296] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10960.120191] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10960.127244] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10960.130644] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10960.134345] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10960.138301] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10960.142281] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10960.147436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10960.153164] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10960.161562] Call Trace: [10960.162662] [10960.163723] ? show_regs.cold.9+0x22/0x2f [10960.165982] ? watchdog_timer_fn.cold.6+0x62/0x10a [10960.168495] ? __hrtimer_run_queues+0x1c2/0x4d0 [10960.174771] ? watchdog+0x40/0x40 [10960.176688] ? hrtimer_interrupt+0x128/0x2c0 [10960.179260] ? smp_apic_timer_interrupt+0xa6/0x270 [10960.181635] ? apic_timer_interrupt+0xf/0x20 [10960.183877] [10960.185026] ? native_safe_halt+0x2e/0x50 [10960.187405] kvm_wait+0xcd/0xe0 [10960.189169] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10960.191792] do_raw_spin_lock+0x177/0x190 [10960.196191] _raw_spin_lock+0x21/0x30 [10960.197806] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10960.200652] ? do_raw_spin_unlock+0x75/0x190 [10960.202351] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10960.205368] ? do_wait_intr+0xf0/0xf0 [10960.206719] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10960.209095] mdc_close+0x53b/0xd70 [mdc] [10960.210964] lmv_close+0x18d/0x490 [lmv] [10960.212458] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10960.214609] ll_md_real_close+0xa5/0x2a0 [lustre] [10960.216970] ll_md_close+0x274/0x800 [lustre] [10960.219375] ll_file_release+0x128/0xc50 [lustre] [10960.221316] __fput+0x10d/0x440 [10960.222547] ____fput+0x12/0x20 [10960.224001] task_work_run+0xc6/0x110 [10960.225490] exit_to_usermode_loop+0x1dd/0x1f0 [10960.227213] do_syscall_64+0x3d6/0x440 [10960.228519] entry_SYSCALL_64_after_hwframe+0x49/0xae [10960.230411] RIP: 0033:0x7f1a63983cd8 [10960.232438] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10960.239395] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10960.243725] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10960.246650] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10960.249492] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10960.252885] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10960.255966] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10968.053382] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10968.060077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10968.094970] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10968.101221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10968.105064] RIP: 0010:native_safe_halt+0x2e/0x50 [10968.108179] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10968.125256] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10968.132182] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10968.136104] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [10968.139251] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10968.145402] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10968.149597] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10968.151863] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10968.161021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10968.165268] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10968.168500] Call Trace: [10968.169827] [10968.170986] ? show_regs.cold.9+0x22/0x2f [10968.173091] ? watchdog_timer_fn.cold.6+0x62/0x10a [10968.175920] ? __hrtimer_run_queues+0x1c2/0x4d0 [10968.178448] ? watchdog+0x40/0x40 [10968.180015] ? hrtimer_interrupt+0x128/0x2c0 [10968.181836] ? smp_apic_timer_interrupt+0xa6/0x270 [10968.183072] ? apic_timer_interrupt+0xf/0x20 [10968.184868] [10968.185665] ? native_safe_halt+0x2e/0x50 [10968.188685] kvm_wait+0xcd/0xe0 [10968.189398] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10968.193925] do_raw_spin_lock+0x177/0x190 [10968.195297] _raw_spin_lock+0x21/0x30 [10968.196134] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10968.198952] ? do_raw_spin_unlock+0x75/0x190 [10968.200754] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10968.202777] ? do_wait_intr+0xf0/0xf0 [10968.203908] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10968.205638] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10968.207757] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10968.209468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10968.213511] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10968.215999] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10968.218150] mdc_intent_lock+0x353/0x7e0 [mdc] [10968.219381] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10968.226056] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10968.228972] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10968.231782] lmv_intent_open+0x338/0x1150 [lmv] [10968.233690] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10968.236257] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10968.238393] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10968.243018] ll_intent_lock+0xc5/0xaa0 [lustre] [10968.245096] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10968.247184] ll_lookup_it+0xe0e/0x2270 [lustre] [10968.249293] ? do_raw_spin_unlock+0x75/0x190 [10968.250792] ? _raw_spin_unlock+0x12/0x30 [10968.252116] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10968.253732] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10968.256035] ll_atomic_open+0x36a/0xd70 [lustre] [10968.259942] lookup_open+0x356/0xb40 [10968.261030] do_last+0x313/0xfc0 [10968.263153] ? nd_jump_root+0xe5/0x160 [10968.264687] ? path_init+0x437/0x520 [10968.266227] path_openat+0xf7/0x500 [10968.267155] ? do_raw_spin_unlock+0x75/0x190 [10968.268439] do_filp_open+0x99/0x140 [10968.270062] ? getname_flags+0x6e/0x330 [10968.273716] ? __check_object_size+0xff/0x256 [10968.275369] ? do_raw_spin_unlock+0x75/0x190 [10968.279166] ? _raw_spin_unlock+0x12/0x30 [10968.282449] do_sys_openat2+0x2b4/0x410 [10968.284699] do_sys_open+0x73/0xa0 [10968.286310] __x64_sys_openat+0x24/0x30 [10968.288323] do_syscall_64+0xc1/0x440 [10968.289959] entry_SYSCALL_64_after_hwframe+0x49/0xae [10968.293051] RIP: 0033:0x7f1a63983332 [10968.294835] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10968.302521] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10968.305400] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10968.308308] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10968.311209] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10968.313913] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10968.316397] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [10971.621714] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8948s! [10971.626039] Showing busy workqueues and worker pools: [10971.628817] workqueue events: flags=0x0 [10971.633013] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10971.639140] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [10971.647695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10971.653157] pending: sec_gc_main [ptlrpc] [10971.658075] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10971.666436] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [10971.675814] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10971.682081] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [10971.689344] workqueue events_long: flags=0x0 [10971.697380] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10971.707929] pending: lu_objects_flush [obdclass] [10971.715754] workqueue events_unbound: flags=0x2 [10971.720444] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10971.728218] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10971.736336] workqueue events_power_efficient: flags=0x80 [10971.743585] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10971.748199] pending: neigh_periodic_work, do_cache_clean [10971.752702] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10971.760700] pending: neigh_periodic_work, check_lifetime [10971.766432] workqueue mm_percpu_wq: flags=0x8 [10971.771900] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10971.778053] pending: vmstat_update [10971.780521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10971.786971] pending: vmstat_update [10971.788399] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10971.791238] pending: vmstat_update [10971.793351] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10971.800250] pending: vmstat_update [10971.804024] workqueue writeback: flags=0x4a [10971.807409] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10971.814426] pending: wb_workfn, wb_workfn [10971.818057] workqueue xprtiod: flags=0xa [10971.823094] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10971.830217] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [10971.838287] workqueue ipv6_addrconf: flags=0x40008 [10971.844752] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10971.850181] pending: addrconf_verify_work [10971.853120] workqueue knbd0-recv: flags=0x1a [10971.856363] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10971.858967] in-flight: 647:recv_work [10971.864319] workqueue ptlrpc_pinger: flags=0x2 [10971.870339] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10971.876939] pending: ptlrpc_pinger_main [ptlrpc] [10971.886306] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10961s workers=2 idle: 652 [10984.059311] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [10984.063769] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [10984.076101] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10984.076178] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10984.076186] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10984.076189] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10984.076203] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10984.076208] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10984.076212] RAX: 0000000000006bc9 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10984.076214] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [10984.076216] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [10984.076218] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [10984.076220] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [10984.076223] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [10984.076226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10984.076228] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [10984.076238] Call Trace: [10984.076243] [10984.076250] ? show_regs.cold.9+0x22/0x2f [10984.084927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10984.085009] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10984.085015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10984.085018] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10984.085034] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10984.085040] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10984.085045] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [10984.085047] RDX: 0000000000001b62 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [10984.085050] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [10984.086336] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [10984.086339] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10984.086342] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [10984.086346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10984.086348] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [10984.086358] Call Trace: [10984.086602] [10984.086607] ? show_regs.cold.9+0x22/0x2f [10984.086620] ? watchdog_timer_fn.cold.6+0x62/0x10a [10984.086626] ? __hrtimer_run_queues+0x1c2/0x4d0 [10984.086630] ? watchdog+0x40/0x40 [10984.086711] ? hrtimer_interrupt+0x128/0x2c0 [10984.086715] ? smp_apic_timer_interrupt+0xa6/0x270 [10984.086724] ? apic_timer_interrupt+0xf/0x20 [10984.086729] [10984.086732] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10984.086740] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10984.086744] do_raw_spin_lock+0x177/0x190 [10984.086749] _raw_spin_lock+0x21/0x30 [10984.086755] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10984.087841] obd_mod_free+0x46/0x140 [ptlrpc] [10984.088786] mdc_commit_open+0x91/0xb0 [mdc] [10984.088825] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10984.137148] ? watchdog_timer_fn.cold.6+0x62/0x10a [10984.137169] ? __hrtimer_run_queues+0x1c2/0x4d0 [10984.137174] ? watchdog+0x40/0x40 [10984.137179] ? hrtimer_interrupt+0x128/0x2c0 [10984.137183] ? smp_apic_timer_interrupt+0xa6/0x270 [10984.148125] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10984.149020] after_reply+0xe2c/0x1780 [ptlrpc] [10984.149250] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10984.150578] ? __next_timer_interrupt+0x160/0x160 [10984.150738] ptlrpcd+0xa82/0xe40 [ptlrpc] [10984.151203] ? do_wait_intr+0xf0/0xf0 [10984.151213] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10984.151880] kthread+0x1d1/0x200 [10984.151887] ? set_kthread_struct+0x70/0x70 [10984.151892] ret_from_fork+0x1f/0x30 [10984.620168] ? apic_timer_interrupt+0xf/0x20 [10984.622209] [10984.623286] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10984.629345] do_raw_spin_lock+0x177/0x190 [10984.633838] _raw_spin_lock+0x21/0x30 [10984.637731] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10984.643152] ? do_raw_spin_unlock+0x75/0x190 [10984.646582] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10984.649076] ? do_wait_intr+0xf0/0xf0 [10984.650617] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10984.653340] mdc_close+0x53b/0xd70 [mdc] [10984.655204] lmv_close+0x18d/0x490 [lmv] [10984.657019] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10984.659519] ll_md_real_close+0xa5/0x2a0 [lustre] [10984.661005] ll_md_close+0x274/0x800 [lustre] [10984.662754] ll_file_release+0x128/0xc50 [lustre] [10984.664241] __fput+0x10d/0x440 [10984.665268] ____fput+0x12/0x20 [10984.666370] task_work_run+0xc6/0x110 [10984.667657] exit_to_usermode_loop+0x1dd/0x1f0 [10984.668963] do_syscall_64+0x3d6/0x440 [10984.670109] entry_SYSCALL_64_after_hwframe+0x49/0xae [10984.671774] RIP: 0033:0x7f1a63983cd8 [10984.672878] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10984.678691] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10984.681004] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [10984.683623] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10984.685949] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [10984.688262] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [10984.690357] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10988.056025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [10988.061893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10988.101644] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10988.108117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10988.114430] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10988.120047] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10988.141121] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10988.146776] RAX: 000000000000465d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10988.154020] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [10988.160871] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [10988.166100] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [10988.171366] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [10988.178140] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [10988.185224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10988.188191] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [10988.193256] Call Trace: [10988.195307] [10988.200002] ? show_regs.cold.9+0x22/0x2f [10988.202330] ? watchdog_timer_fn.cold.6+0x62/0x10a [10988.207410] ? __hrtimer_run_queues+0x1c2/0x4d0 [10988.212432] ? watchdog+0x40/0x40 [10988.216072] ? hrtimer_interrupt+0x128/0x2c0 [10988.217848] ? smp_apic_timer_interrupt+0xa6/0x270 [10988.219226] ? apic_timer_interrupt+0xf/0x20 [10988.222160] [10988.223183] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10988.225128] do_raw_spin_lock+0x177/0x190 [10988.230767] _raw_spin_lock+0x21/0x30 [10988.232240] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10988.238303] ? do_raw_spin_unlock+0x75/0x190 [10988.240014] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10988.245101] ? do_wait_intr+0xf0/0xf0 [10988.248737] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10988.252820] mdc_close+0x53b/0xd70 [mdc] [10988.254136] lmv_close+0x18d/0x490 [lmv] [10988.258241] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10988.265137] ll_md_real_close+0xa5/0x2a0 [lustre] [10988.266869] ll_md_close+0x274/0x800 [lustre] [10988.269056] ll_file_release+0x128/0xc50 [lustre] [10988.271239] __fput+0x10d/0x440 [10988.273892] ____fput+0x12/0x20 [10988.280950] task_work_run+0xc6/0x110 [10988.283317] exit_to_usermode_loop+0x1dd/0x1f0 [10988.292611] do_syscall_64+0x3d6/0x440 [10988.293785] entry_SYSCALL_64_after_hwframe+0x49/0xae [10988.298209] RIP: 0033:0x7f1a63983cd8 [10988.301585] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10988.316667] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10988.321886] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [10988.329167] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [10988.332548] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [10988.338131] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [10988.341120] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10996.056362] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [10996.068847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10996.116165] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10996.130279] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10996.144957] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10996.152429] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10996.167123] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10996.174144] RAX: 0000000000001111 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [10996.177993] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [10996.182445] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [10996.186360] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [10996.189945] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [10996.194086] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [10996.198099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10996.202018] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [10996.209360] Call Trace: [10996.210592] [10996.214415] ? show_regs.cold.9+0x22/0x2f [10996.216339] ? watchdog_timer_fn.cold.6+0x62/0x10a [10996.222803] ? __hrtimer_run_queues+0x1c2/0x4d0 [10996.227630] ? watchdog+0x40/0x40 [10996.232196] ? hrtimer_interrupt+0x128/0x2c0 [10996.236000] ? smp_apic_timer_interrupt+0xa6/0x270 [10996.239202] ? apic_timer_interrupt+0xf/0x20 [10996.241820] [10996.242777] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10996.245350] do_raw_spin_lock+0x177/0x190 [10996.247219] _raw_spin_lock+0x21/0x30 [10996.250939] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10996.254161] ? do_raw_spin_unlock+0x75/0x190 [10996.256152] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10996.260510] ? do_wait_intr+0xf0/0xf0 [10996.262764] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10996.266920] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10996.270088] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10996.273871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10996.277683] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10996.287162] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10996.293718] mdc_intent_lock+0x353/0x7e0 [mdc] [10996.298536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10996.312059] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10996.320106] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10996.327184] lmv_intent_open+0x338/0x1150 [lmv] [10996.335250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10996.343764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10996.352647] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10996.357237] ll_intent_lock+0xc5/0xaa0 [lustre] [10996.365192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10996.368045] ll_lookup_it+0xe0e/0x2270 [lustre] [10996.370669] ? do_raw_spin_unlock+0x75/0x190 [10996.372310] ? _raw_spin_unlock+0x12/0x30 [10996.373761] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10996.375966] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10996.377921] ll_atomic_open+0x36a/0xd70 [lustre] [10996.379303] lookup_open+0x356/0xb40 [10996.380446] do_last+0x313/0xfc0 [10996.381582] ? nd_jump_root+0xe5/0x160 [10996.383196] ? path_init+0x437/0x520 [10996.384754] path_openat+0xf7/0x500 [10996.390059] ? do_raw_spin_unlock+0x75/0x190 [10996.399803] do_filp_open+0x99/0x140 [10996.401721] ? getname_flags+0x6e/0x330 [10996.403985] ? __check_object_size+0xff/0x256 [10996.408376] ? do_raw_spin_unlock+0x75/0x190 [10996.420061] ? _raw_spin_unlock+0x12/0x30 [10996.421583] do_sys_openat2+0x2b4/0x410 [10996.422932] do_sys_open+0x73/0xa0 [10996.432917] __x64_sys_openat+0x24/0x30 [10996.438625] do_syscall_64+0xc1/0x440 [10996.443024] entry_SYSCALL_64_after_hwframe+0x49/0xae [10996.446187] RIP: 0033:0x7f1a63983332 [10996.449830] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10996.468241] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10996.471832] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [10996.484171] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [10996.490217] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [10996.493522] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [10996.500417] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11002.336295] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8979s! [11002.340806] Showing busy workqueues and worker pools: [11002.342898] workqueue events: flags=0x0 [11002.345839] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11002.351081] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11002.356614] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11002.361876] pending: sec_gc_main [ptlrpc] [11002.365044] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11002.368939] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11002.378565] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11002.382730] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11002.387592] workqueue events_long: flags=0x0 [11002.390786] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11002.394735] pending: lu_objects_flush [obdclass] [11002.398130] workqueue events_unbound: flags=0x2 [11002.401787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11002.405698] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11002.415121] workqueue events_power_efficient: flags=0x80 [11002.420563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11002.425083] pending: neigh_periodic_work, do_cache_clean [11002.429991] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11002.435198] pending: neigh_periodic_work, check_lifetime [11002.438308] workqueue mm_percpu_wq: flags=0x8 [11002.441045] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11002.445689] pending: vmstat_update [11002.448015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11002.453814] pending: vmstat_update [11002.457196] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11002.461596] pending: vmstat_update [11002.463858] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11002.467408] pending: vmstat_update [11002.471121] workqueue writeback: flags=0x4a [11002.473531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11002.477383] pending: wb_workfn, wb_workfn [11002.480030] workqueue xprtiod: flags=0xa [11002.482346] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11002.486659] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11002.490945] workqueue ipv6_addrconf: flags=0x40008 [11002.494182] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11002.498157] pending: addrconf_verify_work [11002.501109] workqueue knbd0-recv: flags=0x1a [11002.503357] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11002.506479] in-flight: 647:recv_work [11002.508616] workqueue ptlrpc_pinger: flags=0x2 [11002.510925] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11002.514916] pending: ptlrpc_pinger_main [ptlrpc] [11002.518864] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10992s workers=2 idle: 652 [11012.057049] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11012.060367] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11012.063299] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11012.079410] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11012.079424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11012.079427] RIP: 0010:native_safe_halt+0x2e/0x50 [11012.079441] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11012.079444] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11012.079450] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11012.079452] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [11012.079454] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11012.079456] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11012.079459] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11012.079462] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11012.079465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11012.079467] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11012.079475] Call Trace: [11012.079481] [11012.079739] ? show_regs.cold.9+0x22/0x2f [11012.079750] ? watchdog_timer_fn.cold.6+0x62/0x10a [11012.079760] ? __hrtimer_run_queues+0x1c2/0x4d0 [11012.079765] ? watchdog+0x40/0x40 [11012.079770] ? hrtimer_interrupt+0x128/0x2c0 [11012.079774] ? smp_apic_timer_interrupt+0xa6/0x270 [11012.079781] ? apic_timer_interrupt+0xf/0x20 [11012.079790] [11012.079794] ? native_safe_halt+0x2e/0x50 [11012.079797] kvm_wait+0xcd/0xe0 [11012.079806] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11012.079834] do_raw_spin_lock+0x177/0x190 [11012.079840] _raw_spin_lock+0x21/0x30 [11012.079845] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11012.080151] ? do_raw_spin_unlock+0x75/0x190 [11012.080157] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11012.080529] ? do_wait_intr+0xf0/0xf0 [11012.080538] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11012.080756] mdc_close+0x53b/0xd70 [mdc] [11012.080799] lmv_close+0x18d/0x490 [lmv] [11012.080838] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11012.080973] ll_md_real_close+0xa5/0x2a0 [lustre] [11012.081068] ll_md_close+0x274/0x800 [lustre] [11012.081174] ll_file_release+0x128/0xc50 [lustre] [11012.081445] __fput+0x10d/0x440 [11012.081455] ____fput+0x12/0x20 [11012.081459] task_work_run+0xc6/0x110 [11012.081466] exit_to_usermode_loop+0x1dd/0x1f0 [11012.081473] do_syscall_64+0x3d6/0x440 [11012.081477] entry_SYSCALL_64_after_hwframe+0x49/0xae [11012.081484] RIP: 0033:0x7f1a63983cd8 [11012.081492] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11012.081495] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11012.081500] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11012.081503] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11012.081505] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11012.081508] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11012.081511] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11012.231735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11012.251518] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11012.256440] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11012.260496] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11012.263070] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11012.271619] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11012.274729] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11012.277635] RDX: 0000000000001be5 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11012.280558] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11012.283527] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11012.286228] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11012.289401] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11012.293307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11012.295569] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11012.298654] Call Trace: [11012.300041] [11012.300863] ? show_regs.cold.9+0x22/0x2f [11012.302824] ? watchdog_timer_fn.cold.6+0x62/0x10a [11012.304909] ? __hrtimer_run_queues+0x1c2/0x4d0 [11012.307085] ? watchdog+0x40/0x40 [11012.308871] ? hrtimer_interrupt+0x128/0x2c0 [11012.310757] ? smp_apic_timer_interrupt+0xa6/0x270 [11012.313093] ? apic_timer_interrupt+0xf/0x20 [11012.314911] [11012.315872] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11012.318608] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11012.321043] do_raw_spin_lock+0x177/0x190 [11012.323017] _raw_spin_lock+0x21/0x30 [11012.324674] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11012.328362] obd_mod_free+0x46/0x140 [ptlrpc] [11012.331295] mdc_commit_open+0x91/0xb0 [mdc] [11012.333507] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11012.336013] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11012.338731] after_reply+0xe2c/0x1780 [ptlrpc] [11012.341321] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11012.344055] ? __next_timer_interrupt+0x160/0x160 [11012.346528] ptlrpcd+0xa82/0xe40 [ptlrpc] [11012.348960] ? do_wait_intr+0xf0/0xf0 [11012.351022] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11012.354183] kthread+0x1d1/0x200 [11012.355783] ? set_kthread_struct+0x70/0x70 [11012.358387] ret_from_fork+0x1f/0x30 [11016.056254] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11016.061172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11016.083541] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11016.088927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11016.093396] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11016.096162] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11016.104692] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11016.107983] RAX: 0000000000001435 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11016.111482] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11016.114800] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11016.118031] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11016.121112] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11016.124257] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11016.127743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11016.130362] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11016.134583] Call Trace: [11016.136758] [11016.139113] ? show_regs.cold.9+0x22/0x2f [11016.140717] ? watchdog_timer_fn.cold.6+0x62/0x10a [11016.143443] ? __hrtimer_run_queues+0x1c2/0x4d0 [11016.145196] ? watchdog+0x40/0x40 [11016.147469] ? hrtimer_interrupt+0x128/0x2c0 [11016.149211] ? smp_apic_timer_interrupt+0xa6/0x270 [11016.152045] ? apic_timer_interrupt+0xf/0x20 [11016.154149] [11016.155064] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11016.157567] do_raw_spin_lock+0x177/0x190 [11016.159184] _raw_spin_lock+0x21/0x30 [11016.160702] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11016.162877] ? do_raw_spin_unlock+0x75/0x190 [11016.164700] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11016.166846] ? do_wait_intr+0xf0/0xf0 [11016.168718] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11016.171150] mdc_close+0x53b/0xd70 [mdc] [11016.172857] lmv_close+0x18d/0x490 [lmv] [11016.174785] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11016.177344] ll_md_real_close+0xa5/0x2a0 [lustre] [11016.179421] ll_md_close+0x274/0x800 [lustre] [11016.181542] ll_file_release+0x128/0xc50 [lustre] [11016.183952] __fput+0x10d/0x440 [11016.185177] ____fput+0x12/0x20 [11016.186551] task_work_run+0xc6/0x110 [11016.188027] exit_to_usermode_loop+0x1dd/0x1f0 [11016.189997] do_syscall_64+0x3d6/0x440 [11016.191650] entry_SYSCALL_64_after_hwframe+0x49/0xae [11016.193867] RIP: 0033:0x7f1a63983cd8 [11016.196047] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11016.211793] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11016.216464] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11016.219763] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11016.224063] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11016.228233] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11016.232496] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11028.053063] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11028.058723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11028.103386] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11028.109927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11028.116707] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11028.120254] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11028.127391] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11028.130457] RAX: 0000000000007b0a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11028.135441] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11028.141321] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11028.146949] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11028.150554] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11028.161230] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11028.175375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11028.180719] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11028.184445] Call Trace: [11028.185535] [11028.186492] ? show_regs.cold.9+0x22/0x2f [11028.188565] ? watchdog_timer_fn.cold.6+0x62/0x10a [11028.191726] ? __hrtimer_run_queues+0x1c2/0x4d0 [11028.193669] ? watchdog+0x40/0x40 [11028.195162] ? hrtimer_interrupt+0x128/0x2c0 [11028.199398] ? smp_apic_timer_interrupt+0xa6/0x270 [11028.202624] ? apic_timer_interrupt+0xf/0x20 [11028.205390] [11028.207218] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11028.210191] do_raw_spin_lock+0x177/0x190 [11028.211947] _raw_spin_lock+0x21/0x30 [11028.218294] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11028.223662] ? do_raw_spin_unlock+0x75/0x190 [11028.224891] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11028.226989] ? do_wait_intr+0xf0/0xf0 [11028.229786] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11028.231703] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11028.235449] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11028.239301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11028.241685] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11028.251543] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11028.258318] mdc_intent_lock+0x353/0x7e0 [mdc] [11028.263497] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11028.271369] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11028.282336] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11028.291838] lmv_intent_open+0x338/0x1150 [lmv] [11028.294231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11028.297560] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11028.300590] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11028.302832] ll_intent_lock+0xc5/0xaa0 [lustre] [11028.305671] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11028.307793] ll_lookup_it+0xe0e/0x2270 [lustre] [11028.309916] ? do_raw_spin_unlock+0x75/0x190 [11028.311936] ? _raw_spin_unlock+0x12/0x30 [11028.313856] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11028.316151] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11028.318455] ll_atomic_open+0x36a/0xd70 [lustre] [11028.321810] lookup_open+0x356/0xb40 [11028.324044] do_last+0x313/0xfc0 [11028.325442] ? nd_jump_root+0xe5/0x160 [11028.327742] ? path_init+0x437/0x520 [11028.330405] path_openat+0xf7/0x500 [11028.331876] ? do_raw_spin_unlock+0x75/0x190 [11028.333911] do_filp_open+0x99/0x140 [11028.335922] ? getname_flags+0x6e/0x330 [11028.337176] ? __check_object_size+0xff/0x256 [11028.339753] ? do_raw_spin_unlock+0x75/0x190 [11028.341679] ? _raw_spin_unlock+0x12/0x30 [11028.344386] do_sys_openat2+0x2b4/0x410 [11028.345700] do_sys_open+0x73/0xa0 [11028.346820] __x64_sys_openat+0x24/0x30 [11028.348080] do_syscall_64+0xc1/0x440 [11028.350754] entry_SYSCALL_64_after_hwframe+0x49/0xae [11028.353358] RIP: 0033:0x7f1a63983332 [11028.354785] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11028.363143] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11028.367495] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11028.377133] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11028.379777] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11028.382617] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11028.386974] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11032.550199] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9009s! [11032.560461] Showing busy workqueues and worker pools: [11032.563898] workqueue events: flags=0x0 [11032.565358] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11032.573133] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11032.580697] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11032.586969] pending: sec_gc_main [ptlrpc] [11032.594603] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11032.600989] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11032.608371] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11032.614403] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11032.627098] workqueue events_long: flags=0x0 [11032.632099] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11032.639470] pending: lu_objects_flush [obdclass] [11032.644617] workqueue events_unbound: flags=0x2 [11032.650359] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11032.658969] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11032.675102] workqueue events_power_efficient: flags=0x80 [11032.679699] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11032.685802] pending: neigh_periodic_work, do_cache_clean [11032.695927] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11032.707885] pending: neigh_periodic_work, check_lifetime [11032.715163] workqueue mm_percpu_wq: flags=0x8 [11032.716879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11032.723514] pending: vmstat_update [11032.730118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11032.732667] pending: vmstat_update [11032.738589] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11032.747166] pending: vmstat_update [11032.752593] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11032.765579] pending: vmstat_update [11032.768224] workqueue writeback: flags=0x4a [11032.773139] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11032.783296] pending: wb_workfn, wb_workfn [11032.790386] workqueue xprtiod: flags=0xa [11032.795209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11032.798832] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11032.809310] workqueue ipv6_addrconf: flags=0x40008 [11032.817334] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11032.826828] pending: addrconf_verify_work [11032.831331] workqueue knbd0-recv: flags=0x1a [11032.842893] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11032.849549] in-flight: 647:recv_work [11032.850990] workqueue ptlrpc_pinger: flags=0x2 [11032.864755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11032.877239] pending: ptlrpc_pinger_main [ptlrpc] [11032.892683] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11022s workers=2 idle: 652 [11040.058062] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11040.059462] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11040.067528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11040.067604] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11040.067611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11040.067614] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11040.067628] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11040.067631] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11040.067635] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11040.067638] RDX: 0000000000006fbe RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11040.067640] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11040.067642] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11040.067644] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11040.067646] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11040.067649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11040.067651] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11040.067660] Call Trace: [11040.067665] [11040.067671] ? show_regs.cold.9+0x22/0x2f [11040.067681] ? watchdog_timer_fn.cold.6+0x62/0x10a [11040.067687] ? __hrtimer_run_queues+0x1c2/0x4d0 [11040.067692] ? watchdog+0x40/0x40 [11040.067697] ? hrtimer_interrupt+0x128/0x2c0 [11040.067701] ? smp_apic_timer_interrupt+0xa6/0x270 [11040.067709] ? apic_timer_interrupt+0xf/0x20 [11040.067714] [11040.067716] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11040.067721] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11040.067725] do_raw_spin_lock+0x177/0x190 [11040.067729] _raw_spin_lock+0x21/0x30 [11040.067734] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11040.068119] obd_mod_free+0x46/0x140 [ptlrpc] [11040.068361] mdc_commit_open+0x91/0xb0 [mdc] [11040.068400] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11040.068585] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11040.068782] after_reply+0xe2c/0x1780 [ptlrpc] [11040.071347] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11040.071553] ? __next_timer_interrupt+0x160/0x160 [11040.071565] ptlrpcd+0xa82/0xe40 [ptlrpc] [11040.071770] ? do_wait_intr+0xf0/0xf0 [11040.071780] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11040.072283] kthread+0x1d1/0x200 [11040.072290] ? set_kthread_struct+0x70/0x70 [11040.072294] ret_from_fork+0x1f/0x30 [11040.182513] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11040.199138] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11040.204062] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11040.207766] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11040.210233] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11040.215937] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11040.218109] RAX: 0000000000007d10 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11040.220208] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11040.222309] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11040.224274] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11040.226222] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11040.228291] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11040.230729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11040.232464] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11040.234500] Call Trace: [11040.235215] [11040.235758] ? show_regs.cold.9+0x22/0x2f [11040.237052] ? watchdog_timer_fn.cold.6+0x62/0x10a [11040.238567] ? __hrtimer_run_queues+0x1c2/0x4d0 [11040.239793] ? watchdog+0x40/0x40 [11040.240759] ? hrtimer_interrupt+0x128/0x2c0 [11040.242045] ? smp_apic_timer_interrupt+0xa6/0x270 [11040.243450] ? apic_timer_interrupt+0xf/0x20 [11040.245375] [11040.246003] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11040.247539] do_raw_spin_lock+0x177/0x190 [11040.248679] _raw_spin_lock+0x21/0x30 [11040.249858] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11040.251794] ? do_raw_spin_unlock+0x75/0x190 [11040.253441] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11040.255401] ? do_wait_intr+0xf0/0xf0 [11040.256823] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11040.258725] mdc_close+0x53b/0xd70 [mdc] [11040.260069] lmv_close+0x18d/0x490 [lmv] [11040.261581] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11040.263525] ll_md_real_close+0xa5/0x2a0 [lustre] [11040.265059] ll_md_close+0x274/0x800 [lustre] [11040.266483] ll_file_release+0x128/0xc50 [lustre] [11040.268165] __fput+0x10d/0x440 [11040.269295] ____fput+0x12/0x20 [11040.270389] task_work_run+0xc6/0x110 [11040.271701] exit_to_usermode_loop+0x1dd/0x1f0 [11040.275518] do_syscall_64+0x3d6/0x440 [11040.276817] entry_SYSCALL_64_after_hwframe+0x49/0xae [11040.278584] RIP: 0033:0x7f1a63983cd8 [11040.279808] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11040.286936] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11040.289425] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11040.291589] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11040.293397] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11040.295950] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11040.298129] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11044.059172] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11044.063139] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11044.095379] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11044.101068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11044.115228] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11044.121799] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11044.149838] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11044.155200] RAX: 0000000000007f87 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11044.160174] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11044.164643] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11044.167288] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11044.171001] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11044.178160] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11044.188670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11044.191990] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11044.195354] Call Trace: [11044.196432] [11044.198222] ? show_regs.cold.9+0x22/0x2f [11044.207240] ? watchdog_timer_fn.cold.6+0x62/0x10a [11044.209623] ? __hrtimer_run_queues+0x1c2/0x4d0 [11044.214002] ? watchdog+0x40/0x40 [11044.217845] ? hrtimer_interrupt+0x128/0x2c0 [11044.222449] ? smp_apic_timer_interrupt+0xa6/0x270 [11044.229969] ? apic_timer_interrupt+0xf/0x20 [11044.234206] [11044.240256] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11044.243444] do_raw_spin_lock+0x177/0x190 [11044.245952] _raw_spin_lock+0x21/0x30 [11044.248949] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11044.258539] ? do_raw_spin_unlock+0x75/0x190 [11044.263951] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11044.268961] ? do_wait_intr+0xf0/0xf0 [11044.272002] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11044.279138] mdc_close+0x53b/0xd70 [mdc] [11044.286500] lmv_close+0x18d/0x490 [lmv] [11044.288252] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11044.289685] ll_md_real_close+0xa5/0x2a0 [lustre] [11044.298346] ll_md_close+0x274/0x800 [lustre] [11044.301924] ll_file_release+0x128/0xc50 [lustre] [11044.305069] __fput+0x10d/0x440 [11044.310723] ____fput+0x12/0x20 [11044.315410] task_work_run+0xc6/0x110 [11044.320553] exit_to_usermode_loop+0x1dd/0x1f0 [11044.322209] do_syscall_64+0x3d6/0x440 [11044.329650] entry_SYSCALL_64_after_hwframe+0x49/0xae [11044.339885] RIP: 0033:0x7f1a63983cd8 [11044.347710] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11044.374107] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11044.382750] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11044.393692] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11044.405501] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11044.411524] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11044.422244] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11060.058201] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11060.063066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11060.106029] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11060.116464] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11060.129674] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11060.138547] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11060.155127] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11060.162819] RAX: 0000000000007f86 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11060.166464] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11060.173246] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11060.179141] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11060.183989] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11060.190573] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11060.194866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11060.203434] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11060.209262] Call Trace: [11060.211050] [11060.212701] ? show_regs.cold.9+0x22/0x2f [11060.215213] ? watchdog_timer_fn.cold.6+0x62/0x10a [11060.229273] ? __hrtimer_run_queues+0x1c2/0x4d0 [11060.235397] ? watchdog+0x40/0x40 [11060.239039] ? hrtimer_interrupt+0x128/0x2c0 [11060.242132] ? smp_apic_timer_interrupt+0xa6/0x270 [11060.247143] ? apic_timer_interrupt+0xf/0x20 [11060.250179] [11060.251213] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11060.259807] do_raw_spin_lock+0x177/0x190 [11060.262477] _raw_spin_lock+0x21/0x30 [11060.264568] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11060.267782] ? do_raw_spin_unlock+0x75/0x190 [11060.269905] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11060.273526] ? do_wait_intr+0xf0/0xf0 [11060.275922] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11060.278597] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11060.281782] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11060.285194] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11060.287906] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11060.290568] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11060.293032] mdc_intent_lock+0x353/0x7e0 [mdc] [11060.295911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11060.298747] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11060.302101] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11060.304303] lmv_intent_open+0x338/0x1150 [lmv] [11060.307533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11060.310120] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11060.313114] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11060.315305] ll_intent_lock+0xc5/0xaa0 [lustre] [11060.318237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11060.320625] ll_lookup_it+0xe0e/0x2270 [lustre] [11060.322895] ? do_raw_spin_unlock+0x75/0x190 [11060.324708] ? _raw_spin_unlock+0x12/0x30 [11060.327031] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11060.329250] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11060.331579] ll_atomic_open+0x36a/0xd70 [lustre] [11060.333775] lookup_open+0x356/0xb40 [11060.334926] do_last+0x313/0xfc0 [11060.336020] ? nd_jump_root+0xe5/0x160 [11060.337754] ? path_init+0x437/0x520 [11060.339431] path_openat+0xf7/0x500 [11060.341090] ? do_raw_spin_unlock+0x75/0x190 [11060.342957] do_filp_open+0x99/0x140 [11060.344797] ? getname_flags+0x6e/0x330 [11060.346397] ? __check_object_size+0xff/0x256 [11060.349271] ? do_raw_spin_unlock+0x75/0x190 [11060.351946] ? _raw_spin_unlock+0x12/0x30 [11060.353965] do_sys_openat2+0x2b4/0x410 [11060.356213] do_sys_open+0x73/0xa0 [11060.358751] __x64_sys_openat+0x24/0x30 [11060.363420] do_syscall_64+0xc1/0x440 [11060.365142] entry_SYSCALL_64_after_hwframe+0x49/0xae [11060.368916] RIP: 0033:0x7f1a63983332 [11060.370558] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11060.382326] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11060.385956] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11060.389663] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11060.392885] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11060.395843] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11060.398755] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11063.268026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9040s! [11063.276328] Showing busy workqueues and worker pools: [11063.281185] workqueue events: flags=0x0 [11063.289865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11063.296209] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11063.307200] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11063.314762] pending: sec_gc_main [ptlrpc] [11063.321137] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11063.331149] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11063.343906] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11063.346148] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11063.369142] workqueue events_long: flags=0x0 [11063.375350] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11063.381369] pending: lu_objects_flush [obdclass] [11063.385890] workqueue events_unbound: flags=0x2 [11063.388828] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11063.393824] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11063.412305] workqueue events_power_efficient: flags=0x80 [11063.414990] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11063.419186] pending: neigh_periodic_work, do_cache_clean [11063.424678] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11063.430984] pending: neigh_periodic_work, check_lifetime [11063.433549] workqueue mm_percpu_wq: flags=0x8 [11063.442873] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11063.446291] pending: vmstat_update [11063.448273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11063.451738] pending: vmstat_update [11063.453811] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11063.458202] pending: vmstat_update [11063.460201] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11063.463729] pending: vmstat_update [11063.465858] workqueue writeback: flags=0x4a [11063.468367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11063.471562] pending: wb_workfn, wb_workfn [11063.473962] workqueue xprtiod: flags=0xa [11063.476085] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11063.479205] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11063.487131] workqueue ipv6_addrconf: flags=0x40008 [11063.489250] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11063.493109] pending: addrconf_verify_work [11063.495458] workqueue knbd0-recv: flags=0x1a [11063.497118] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11063.506375] in-flight: 647:recv_work [11063.508971] workqueue ptlrpc_pinger: flags=0x2 [11063.514496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11063.518199] pending: ptlrpc_pinger_main [ptlrpc] [11063.520752] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11053s workers=2 idle: 652 [11068.057027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11068.058033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11068.059948] Modules linked in: [11068.062480] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11068.062559] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11068.062566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11068.062569] RIP: 0010:native_safe_halt+0x2e/0x50 [11068.062590] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11068.062594] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [11068.062599] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11068.062604] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11068.062606] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11068.062608] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11068.062610] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11068.062614] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11068.062616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11068.062619] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11068.062626] Call Trace: [11068.062631] [11068.062639] ? show_regs.cold.9+0x22/0x2f [11068.062648] ? watchdog_timer_fn.cold.6+0x62/0x10a [11068.062653] ? __hrtimer_run_queues+0x1c2/0x4d0 [11068.062659] ? watchdog+0x40/0x40 [11068.062663] ? hrtimer_interrupt+0x128/0x2c0 [11068.062667] ? smp_apic_timer_interrupt+0xa6/0x270 [11068.062675] ? apic_timer_interrupt+0xf/0x20 [11068.062679] [11068.062682] ? native_safe_halt+0x2e/0x50 [11068.062685] kvm_wait+0xcd/0xe0 [11068.062691] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11068.062697] do_raw_spin_lock+0x177/0x190 [11068.062701] _raw_spin_lock+0x21/0x30 [11068.062705] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11068.063105] obd_mod_free+0x46/0x140 [ptlrpc] [11068.064426] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata [11068.103464] mdc_commit_open+0x91/0xb0 [mdc] [11068.103512] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11068.103994] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11068.104211] after_reply+0xe2c/0x1780 [ptlrpc] [11068.104436] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11068.104657] ? __next_timer_interrupt+0x160/0x160 [11068.104667] ptlrpcd+0xa82/0xe40 [ptlrpc] [11068.104878] ? do_wait_intr+0xf0/0xf0 [11068.104887] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11068.105142] kthread+0x1d1/0x200 [11068.105147] ? set_kthread_struct+0x70/0x70 [11068.105151] ret_from_fork+0x1f/0x30 [11068.364476] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11068.373491] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11068.390393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11068.405336] RIP: 0010:native_safe_halt+0x2e/0x50 [11068.407487] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11068.433632] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11068.437454] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11068.453446] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [11068.463070] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11068.466515] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11068.470302] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11068.473340] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11068.475915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11068.478441] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11068.481920] Call Trace: [11068.482984] [11068.483894] ? show_regs.cold.9+0x22/0x2f [11068.489794] ? watchdog_timer_fn.cold.6+0x62/0x10a [11068.494724] ? __hrtimer_run_queues+0x1c2/0x4d0 [11068.497206] ? watchdog+0x40/0x40 [11068.499182] ? hrtimer_interrupt+0x128/0x2c0 [11068.501053] ? smp_apic_timer_interrupt+0xa6/0x270 [11068.507456] ? apic_timer_interrupt+0xf/0x20 [11068.512424] [11068.514522] ? native_safe_halt+0x2e/0x50 [11068.518920] kvm_wait+0xcd/0xe0 [11068.523536] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11068.525426] do_raw_spin_lock+0x177/0x190 [11068.527769] _raw_spin_lock+0x21/0x30 [11068.529437] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11068.531815] ? do_raw_spin_unlock+0x75/0x190 [11068.533240] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11068.536136] ? do_wait_intr+0xf0/0xf0 [11068.539128] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11068.548269] mdc_close+0x53b/0xd70 [mdc] [11068.549403] lmv_close+0x18d/0x490 [lmv] [11068.553163] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11068.558048] ll_md_real_close+0xa5/0x2a0 [lustre] [11068.560890] ll_md_close+0x274/0x800 [lustre] [11068.566933] ll_file_release+0x128/0xc50 [lustre] [11068.570537] __fput+0x10d/0x440 [11068.571737] ____fput+0x12/0x20 [11068.572944] task_work_run+0xc6/0x110 [11068.577746] exit_to_usermode_loop+0x1dd/0x1f0 [11068.581054] do_syscall_64+0x3d6/0x440 [11068.585060] entry_SYSCALL_64_after_hwframe+0x49/0xae [11068.589436] RIP: 0033:0x7f1a63983cd8 [11068.595175] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11068.601923] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11068.606209] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11068.609323] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11068.612412] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11068.615942] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11068.621384] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11072.056020] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11072.058546] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11072.073518] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11072.077324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11072.080449] RIP: 0010:native_safe_halt+0x2e/0x50 [11072.082444] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11072.089174] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11072.091909] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11072.094771] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11072.097395] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11072.099671] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11072.102228] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11072.104479] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11072.107156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11072.108920] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11072.111368] Call Trace: [11072.112145] [11072.112819] ? show_regs.cold.9+0x22/0x2f [11072.114510] ? watchdog_timer_fn.cold.6+0x62/0x10a [11072.116130] ? __hrtimer_run_queues+0x1c2/0x4d0 [11072.117812] ? watchdog+0x40/0x40 [11072.118856] ? hrtimer_interrupt+0x128/0x2c0 [11072.120146] ? smp_apic_timer_interrupt+0xa6/0x270 [11072.121903] ? apic_timer_interrupt+0xf/0x20 [11072.123999] [11072.124788] ? native_safe_halt+0x2e/0x50 [11072.126147] kvm_wait+0xcd/0xe0 [11072.127239] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11072.129251] do_raw_spin_lock+0x177/0x190 [11072.130631] _raw_spin_lock+0x21/0x30 [11072.131938] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11072.134308] ? do_raw_spin_unlock+0x75/0x190 [11072.135704] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11072.137432] ? do_wait_intr+0xf0/0xf0 [11072.138590] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11072.140726] mdc_close+0x53b/0xd70 [mdc] [11072.142612] lmv_close+0x18d/0x490 [lmv] [11072.144176] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11072.146420] ll_md_real_close+0xa5/0x2a0 [lustre] [11072.148214] ll_md_close+0x274/0x800 [lustre] [11072.150356] ll_file_release+0x128/0xc50 [lustre] [11072.152302] __fput+0x10d/0x440 [11072.153323] ____fput+0x12/0x20 [11072.154396] task_work_run+0xc6/0x110 [11072.155708] exit_to_usermode_loop+0x1dd/0x1f0 [11072.157267] do_syscall_64+0x3d6/0x440 [11072.158483] entry_SYSCALL_64_after_hwframe+0x49/0xae [11072.160038] RIP: 0033:0x7f1a63983cd8 [11072.161437] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11072.167806] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11072.170200] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11072.172406] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11072.175182] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11072.177493] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11072.180132] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11082.654836] rcu: INFO: rcu_sched self-detected stall on CPU [11082.666903] rcu: 3-...!: (5712983 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [11082.670689] (t=9060263 jiffies g=269633 q=3255) [11082.672701] rcu: rcu_sched kthread starved for 9059909 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [11082.677171] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11082.680210] rcu: RCU grace-period kthread stack dump: [11082.682130] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11082.686370] Call Trace: [11082.687201] __schedule+0x351/0xcb0 [11082.688929] schedule+0xc0/0x180 [11082.690049] schedule_timeout+0xb4/0x190 [11082.691408] ? do_raw_spin_unlock+0x75/0x190 [11082.693367] ? __next_timer_interrupt+0x160/0x160 [11082.695316] ? prepare_to_swait_event+0xf3/0x140 [11082.696983] rcu_gp_kthread+0x8dd/0x15b0 [11082.698326] ? force_qs_rnp+0x2c0/0x2c0 [11082.699360] kthread+0x1d1/0x200 [11082.701312] ? set_kthread_struct+0x70/0x70 [11082.702647] ret_from_fork+0x1f/0x30 [11082.703709] rcu: Stack dump where RCU GP kthread last ran: [11082.705250] NMI backtrace for cpu 3 [11082.706175] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11082.709225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11082.712189] Call Trace: [11082.712830] [11082.713441] ? dump_stack+0xbb/0x10e [11082.714600] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11082.715805] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11082.717369] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11082.718826] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11082.720637] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11082.722040] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [11082.723766] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [11082.725209] ? update_process_times+0x6d/0xc0 [11082.726673] ? tick_sched_handle.isra.18+0x27/0x90 [11082.728002] ? tick_sched_timer+0x53/0xb0 [11082.729062] ? __hrtimer_run_queues+0x1c2/0x4d0 [11082.730314] ? tick_sched_handle.isra.18+0x90/0x90 [11082.731813] ? hrtimer_interrupt+0x128/0x2c0 [11082.733079] ? smp_apic_timer_interrupt+0xa6/0x270 [11082.734501] ? apic_timer_interrupt+0xf/0x20 [11082.735719] [11082.736396] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11082.738039] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11082.739648] ? do_raw_spin_lock+0x177/0x190 [11082.740895] ? _raw_spin_lock+0x21/0x30 [11082.742048] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11082.743669] ? obd_mod_free+0x46/0x140 [ptlrpc] [11082.745369] ? mdc_commit_open+0x91/0xb0 [mdc] [11082.746932] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11082.748420] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11082.750107] ? after_reply+0xe2c/0x1780 [ptlrpc] [11082.751520] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11082.753656] ? __next_timer_interrupt+0x160/0x160 [11082.754907] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11082.756516] ? do_wait_intr+0xf0/0xf0 [11082.757371] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11082.758990] ? kthread+0x1d1/0x200 [11082.759954] ? set_kthread_struct+0x70/0x70 [11082.760940] ? ret_from_fork+0x1f/0x30 [11082.761970] NMI backtrace for cpu 3 [11082.763053] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11082.766079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11082.768739] Call Trace: [11082.769405] [11082.770642] ? dump_stack+0xbb/0x10e [11082.771762] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11082.773541] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11082.774702] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11082.778929] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11082.780702] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11082.783089] ? rcu_dump_cpu_stacks+0x11b/0x171 [11082.787848] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11082.789374] ? update_process_times+0x6d/0xc0 [11082.794466] ? tick_sched_handle.isra.18+0x27/0x90 [11082.796224] ? tick_sched_timer+0x53/0xb0 [11082.804513] ? __hrtimer_run_queues+0x1c2/0x4d0 [11082.806218] ? tick_sched_handle.isra.18+0x90/0x90 [11082.814568] ? hrtimer_interrupt+0x128/0x2c0 [11082.815705] ? smp_apic_timer_interrupt+0xa6/0x270 [11082.817320] ? apic_timer_interrupt+0xf/0x20 [11082.825620] [11082.826459] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11082.836475] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11082.837773] ? do_raw_spin_lock+0x177/0x190 [11082.840414] ? _raw_spin_lock+0x21/0x30 [11082.848771] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11082.850321] ? obd_mod_free+0x46/0x140 [ptlrpc] [11082.859501] ? mdc_commit_open+0x91/0xb0 [mdc] [11082.860671] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11082.866210] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11082.870306] ? after_reply+0xe2c/0x1780 [ptlrpc] [11082.877047] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11082.878930] ? __next_timer_interrupt+0x160/0x160 [11082.881865] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11082.883358] ? do_wait_intr+0xf0/0xf0 [11082.884846] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11082.887056] ? kthread+0x1d1/0x200 [11082.889045] ? set_kthread_struct+0x70/0x70 [11082.890202] ? ret_from_fork+0x1f/0x30 [11088.053660] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11088.057045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11088.081980] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11088.088010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11088.092559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11088.096768] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11088.106881] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11088.111362] RAX: 0000000000007113 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11088.119856] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11088.123325] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11088.126662] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11088.131556] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11088.135774] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11088.140600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11088.143585] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11088.148304] Call Trace: [11088.149542] [11088.151220] ? show_regs.cold.9+0x22/0x2f [11088.153405] ? watchdog_timer_fn.cold.6+0x62/0x10a [11088.155558] ? __hrtimer_run_queues+0x1c2/0x4d0 [11088.157700] ? watchdog+0x40/0x40 [11088.159358] ? hrtimer_interrupt+0x128/0x2c0 [11088.161307] ? smp_apic_timer_interrupt+0xa6/0x270 [11088.163771] ? apic_timer_interrupt+0xf/0x20 [11088.165668] [11088.166621] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11088.169461] do_raw_spin_lock+0x177/0x190 [11088.171787] _raw_spin_lock+0x21/0x30 [11088.173640] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11088.176114] ? do_raw_spin_unlock+0x75/0x190 [11088.178928] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11088.181607] ? do_wait_intr+0xf0/0xf0 [11088.182818] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11088.186420] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11088.188919] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11088.191827] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11088.194671] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11088.197270] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11088.199398] mdc_intent_lock+0x353/0x7e0 [mdc] [11088.201756] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11088.204009] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11088.206823] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11088.209212] lmv_intent_open+0x338/0x1150 [lmv] [11088.211277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11088.213536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11088.216173] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11088.218152] ll_intent_lock+0xc5/0xaa0 [lustre] [11088.221507] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11088.223852] ll_lookup_it+0xe0e/0x2270 [lustre] [11088.226263] ? do_raw_spin_unlock+0x75/0x190 [11088.227960] ? _raw_spin_unlock+0x12/0x30 [11088.230585] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11088.232588] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11088.234111] ll_atomic_open+0x36a/0xd70 [lustre] [11088.235652] lookup_open+0x356/0xb40 [11088.236753] do_last+0x313/0xfc0 [11088.237691] ? nd_jump_root+0xe5/0x160 [11088.238841] ? path_init+0x437/0x520 [11088.240150] path_openat+0xf7/0x500 [11088.241356] ? do_raw_spin_unlock+0x75/0x190 [11088.242785] do_filp_open+0x99/0x140 [11088.243900] ? getname_flags+0x6e/0x330 [11088.244942] ? __check_object_size+0xff/0x256 [11088.246057] ? do_raw_spin_unlock+0x75/0x190 [11088.247177] ? _raw_spin_unlock+0x12/0x30 [11088.248273] do_sys_openat2+0x2b4/0x410 [11088.249354] do_sys_open+0x73/0xa0 [11088.250292] __x64_sys_openat+0x24/0x30 [11088.251343] do_syscall_64+0xc1/0x440 [11088.252422] entry_SYSCALL_64_after_hwframe+0x49/0xae [11088.254063] RIP: 0033:0x7f1a63983332 [11088.255457] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11088.261544] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11088.264219] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11088.266361] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11088.268869] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11088.271168] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11088.274112] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11093.984971] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9070s! [11093.992209] Showing busy workqueues and worker pools: [11093.998837] workqueue events: flags=0x0 [11094.001413] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11094.004726] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11094.008803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11094.012052] pending: sec_gc_main [ptlrpc] [11094.014627] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11094.017529] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11094.021984] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11094.024802] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11094.028706] workqueue events_long: flags=0x0 [11094.030414] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11094.032773] pending: lu_objects_flush [obdclass] [11094.035203] workqueue events_unbound: flags=0x2 [11094.037095] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11094.039237] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11094.042473] workqueue events_power_efficient: flags=0x80 [11094.044872] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11094.047667] pending: neigh_periodic_work, do_cache_clean [11094.050508] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11094.053065] pending: neigh_periodic_work, check_lifetime [11094.055725] workqueue mm_percpu_wq: flags=0x8 [11094.057268] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11094.060175] pending: vmstat_update [11094.061763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11094.065107] pending: vmstat_update [11094.066832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11094.069452] pending: vmstat_update [11094.070909] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11094.073636] pending: vmstat_update [11094.075259] workqueue writeback: flags=0x4a [11094.076963] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11094.079598] pending: wb_workfn, wb_workfn [11094.081277] workqueue xprtiod: flags=0xa [11094.082932] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11094.085633] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11094.088927] workqueue ipv6_addrconf: flags=0x40008 [11094.090875] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11094.093862] pending: addrconf_verify_work [11094.095745] workqueue knbd0-recv: flags=0x1a [11094.097468] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11094.100053] in-flight: 647:recv_work [11094.101765] workqueue ptlrpc_pinger: flags=0x2 [11094.103888] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11094.106593] pending: ptlrpc_pinger_main [ptlrpc] [11094.109128] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11084s workers=2 idle: 652 [11096.057067] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11096.060476] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11096.083517] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11096.088550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11096.093051] RIP: 0010:native_safe_halt+0x2e/0x50 [11096.095096] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11096.103217] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11096.106897] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11096.110520] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11096.115270] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11096.122563] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11096.128287] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11096.132113] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11096.143331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11096.145799] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11096.149119] Call Trace: [11096.150004] [11096.151561] ? show_regs.cold.9+0x22/0x2f [11096.153238] ? watchdog_timer_fn.cold.6+0x62/0x10a [11096.155508] ? __hrtimer_run_queues+0x1c2/0x4d0 [11096.159732] ? watchdog+0x40/0x40 [11096.161382] ? hrtimer_interrupt+0x128/0x2c0 [11096.163357] ? smp_apic_timer_interrupt+0xa6/0x270 [11096.165493] ? apic_timer_interrupt+0xf/0x20 [11096.166799] [11096.167542] ? native_safe_halt+0x2e/0x50 [11096.168671] kvm_wait+0xcd/0xe0 [11096.170613] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11096.173109] do_raw_spin_lock+0x177/0x190 [11096.175207] _raw_spin_lock+0x21/0x30 [11096.176912] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11096.181646] ? do_raw_spin_unlock+0x75/0x190 [11096.187731] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11096.192290] ? do_wait_intr+0xf0/0xf0 [11096.194278] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11096.197252] mdc_close+0x53b/0xd70 [mdc] [11096.199290] lmv_close+0x18d/0x490 [lmv] [11096.201163] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11096.204243] ll_md_real_close+0xa5/0x2a0 [lustre] [11096.214629] ll_md_close+0x274/0x800 [lustre] [11096.216491] ll_file_release+0x128/0xc50 [lustre] [11096.218228] __fput+0x10d/0x440 [11096.219847] ____fput+0x12/0x20 [11096.222102] task_work_run+0xc6/0x110 [11096.224459] exit_to_usermode_loop+0x1dd/0x1f0 [11096.227153] do_syscall_64+0x3d6/0x440 [11096.228239] entry_SYSCALL_64_after_hwframe+0x49/0xae [11096.231722] RIP: 0033:0x7f1a63983cd8 [11096.233458] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11096.244823] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11096.248236] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11096.252934] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11096.260603] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11096.265107] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11096.270197] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11100.056014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11100.061091] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11100.083898] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11100.088135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11100.092926] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11100.095504] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11100.101770] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11100.104529] RAX: 000000000000551b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11100.107212] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11100.109703] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11100.112185] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11100.114742] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11100.117397] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11100.120222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11100.122077] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11100.124550] Call Trace: [11100.125359] [11100.125960] ? show_regs.cold.9+0x22/0x2f [11100.127553] ? watchdog_timer_fn.cold.6+0x62/0x10a [11100.129157] ? __hrtimer_run_queues+0x1c2/0x4d0 [11100.131008] ? watchdog+0x40/0x40 [11100.132453] ? hrtimer_interrupt+0x128/0x2c0 [11100.134007] ? smp_apic_timer_interrupt+0xa6/0x270 [11100.135735] ? apic_timer_interrupt+0xf/0x20 [11100.137434] [11100.138192] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11100.140325] do_raw_spin_lock+0x177/0x190 [11100.141852] _raw_spin_lock+0x21/0x30 [11100.143715] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11100.146607] ? do_raw_spin_unlock+0x75/0x190 [11100.148665] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11100.151636] ? do_wait_intr+0xf0/0xf0 [11100.154843] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11100.158030] mdc_close+0x53b/0xd70 [mdc] [11100.161599] lmv_close+0x18d/0x490 [lmv] [11100.164658] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11100.169449] ll_md_real_close+0xa5/0x2a0 [lustre] [11100.173188] ll_md_close+0x274/0x800 [lustre] [11100.178484] ll_file_release+0x128/0xc50 [lustre] [11100.182017] __fput+0x10d/0x440 [11100.183573] ____fput+0x12/0x20 [11100.185153] task_work_run+0xc6/0x110 [11100.187101] exit_to_usermode_loop+0x1dd/0x1f0 [11100.190208] do_syscall_64+0x3d6/0x440 [11100.192337] entry_SYSCALL_64_after_hwframe+0x49/0xae [11100.194024] RIP: 0033:0x7f1a63983cd8 [11100.195058] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11100.204040] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11100.208324] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11100.213669] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11100.218068] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11100.224006] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11100.227605] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11108.058035] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [11108.064233] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11108.121788] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11108.141172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11108.148110] RIP: 0010:native_safe_halt+0x2e/0x50 [11108.151556] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11108.173723] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11108.177989] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11108.183867] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11108.190952] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11108.201863] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11108.208696] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11108.214904] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11108.222559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11108.237829] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11108.244777] Call Trace: [11108.249567] [11108.250679] ? show_regs.cold.9+0x22/0x2f [11108.254866] ? watchdog_timer_fn.cold.6+0x62/0x10a [11108.257974] ? __hrtimer_run_queues+0x1c2/0x4d0 [11108.260857] ? watchdog+0x40/0x40 [11108.265130] ? hrtimer_interrupt+0x128/0x2c0 [11108.269543] ? smp_apic_timer_interrupt+0xa6/0x270 [11108.273631] ? apic_timer_interrupt+0xf/0x20 [11108.275358] [11108.276285] ? native_safe_halt+0x2e/0x50 [11108.279371] kvm_wait+0xcd/0xe0 [11108.280921] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11108.285785] do_raw_spin_lock+0x177/0x190 [11108.291389] _raw_spin_lock+0x21/0x30 [11108.295146] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11108.302741] obd_mod_free+0x46/0x140 [ptlrpc] [11108.313374] mdc_commit_open+0x91/0xb0 [mdc] [11108.320848] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11108.327592] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11108.331448] after_reply+0xe2c/0x1780 [ptlrpc] [11108.333539] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11108.339445] ? __next_timer_interrupt+0x160/0x160 [11108.342708] ptlrpcd+0xa82/0xe40 [ptlrpc] [11108.347291] ? do_wait_intr+0xf0/0xf0 [11108.349980] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11108.360956] kthread+0x1d1/0x200 [11108.362193] ? set_kthread_struct+0x70/0x70 [11108.365452] ret_from_fork+0x1f/0x30 [11120.053181] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11120.057757] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11120.095481] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11120.108767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11120.114448] RIP: 0010:native_safe_halt+0x2e/0x50 [11120.117184] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11120.124881] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11120.129034] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11120.132738] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [11120.135182] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11120.140155] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11120.153519] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11120.161371] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11120.167959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11120.172189] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11120.176380] Call Trace: [11120.177854] [11120.179046] ? show_regs.cold.9+0x22/0x2f [11120.181055] ? watchdog_timer_fn.cold.6+0x62/0x10a [11120.184053] ? __hrtimer_run_queues+0x1c2/0x4d0 [11120.186763] ? watchdog+0x40/0x40 [11120.188173] ? hrtimer_interrupt+0x128/0x2c0 [11120.189274] ? smp_apic_timer_interrupt+0xa6/0x270 [11120.191666] ? apic_timer_interrupt+0xf/0x20 [11120.196283] [11120.198456] ? native_safe_halt+0x2e/0x50 [11120.201310] kvm_wait+0xcd/0xe0 [11120.205162] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11120.210638] do_raw_spin_lock+0x177/0x190 [11120.215926] _raw_spin_lock+0x21/0x30 [11120.219134] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11120.223170] ? do_raw_spin_unlock+0x75/0x190 [11120.227109] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11120.230837] ? do_wait_intr+0xf0/0xf0 [11120.235434] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11120.241021] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11120.248257] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11120.255701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11120.259004] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11120.261250] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11120.269889] mdc_intent_lock+0x353/0x7e0 [mdc] [11120.275462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11120.283279] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11120.291371] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11120.296052] lmv_intent_open+0x338/0x1150 [lmv] [11120.297839] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11120.305895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11120.308977] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11120.316282] ll_intent_lock+0xc5/0xaa0 [lustre] [11120.319872] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11120.325895] ll_lookup_it+0xe0e/0x2270 [lustre] [11120.330089] ? do_raw_spin_unlock+0x75/0x190 [11120.331706] ? _raw_spin_unlock+0x12/0x30 [11120.334695] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11120.336617] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11120.345024] ll_atomic_open+0x36a/0xd70 [lustre] [11120.355272] lookup_open+0x356/0xb40 [11120.362138] do_last+0x313/0xfc0 [11120.363438] ? nd_jump_root+0xe5/0x160 [11120.366166] ? path_init+0x437/0x520 [11120.371357] path_openat+0xf7/0x500 [11120.373995] ? do_raw_spin_unlock+0x75/0x190 [11120.377382] do_filp_open+0x99/0x140 [11120.380445] ? getname_flags+0x6e/0x330 [11120.381881] ? __check_object_size+0xff/0x256 [11120.385953] ? do_raw_spin_unlock+0x75/0x190 [11120.389464] ? _raw_spin_unlock+0x12/0x30 [11120.391519] do_sys_openat2+0x2b4/0x410 [11120.393870] do_sys_open+0x73/0xa0 [11120.398785] __x64_sys_openat+0x24/0x30 [11120.400399] do_syscall_64+0xc1/0x440 [11120.405989] entry_SYSCALL_64_after_hwframe+0x49/0xae [11120.410634] RIP: 0033:0x7f1a63983332 [11120.412584] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11120.423789] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11120.426124] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11120.429361] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11120.432585] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11120.435907] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11120.439208] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11124.061096] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11124.079679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11124.129435] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11124.135375] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11124.141678] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11124.145798] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11124.154884] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11124.160362] RAX: 0000000000000908 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11124.163651] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11124.168554] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11124.171771] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11124.176808] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11124.181125] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11124.186646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11124.190459] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11124.195259] BUG: workqueue lockup - pool [11124.198635] Call Trace: [11124.198650] [11124.198656] ? show_regs.cold.9+0x22/0x2f [11124.198672] ? watchdog_timer_fn.cold.6+0x62/0x10a [11124.198679] ? __hrtimer_run_queues+0x1c2/0x4d0 [11124.198683] ? watchdog+0x40/0x40 [11124.198688] ? hrtimer_interrupt+0x128/0x2c0 [11124.198692] ? smp_apic_timer_interrupt+0xa6/0x270 [11124.198701] ? apic_timer_interrupt+0xf/0x20 [11124.198706] [11124.198709] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11124.198717] do_raw_spin_lock+0x177/0x190 [11124.198722] _raw_spin_lock+0x21/0x30 [11124.198728] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11124.199190] ? do_raw_spin_unlock+0x75/0x190 [11124.199196] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11124.199399] ? do_wait_intr+0xf0/0xf0 [11124.199407] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11124.199620] mdc_close+0x53b/0xd70 [mdc] [11124.199661] lmv_close+0x18d/0x490 [lmv] [11124.199693] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11124.199810] ll_md_real_close+0xa5/0x2a0 [lustre] [11124.199891] ll_md_close+0x274/0x800 [lustre] [11124.199977] ll_file_release+0x128/0xc50 [lustre] [11124.203968] cpus=0-3 [11124.206151] __fput+0x10d/0x440 [11124.206169] ____fput+0x12/0x20 [11124.206174] task_work_run+0xc6/0x110 [11124.206182] exit_to_usermode_loop+0x1dd/0x1f0 [11124.206190] do_syscall_64+0x3d6/0x440 [11124.206195] entry_SYSCALL_64_after_hwframe+0x49/0xae [11124.206203] RIP: 0033:0x7f1a63983cd8 [11124.206210] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11124.206214] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11124.206220] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11124.206222] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11124.206225] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11124.206227] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11124.206229] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11124.408785] flags=0x4 nice=0 stuck for 9101s! [11124.410667] Showing busy workqueues and worker pools: [11124.412963] workqueue events: flags=0x0 [11124.416270] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11124.421251] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11124.423465] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11124.426573] pending: sec_gc_main [ptlrpc] [11124.431090] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11124.437896] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11124.448459] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11124.451974] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11124.460974] workqueue events_long: flags=0x0 [11124.464142] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11124.468135] pending: lu_objects_flush [obdclass] [11124.470944] workqueue events_unbound: flags=0x2 [11124.472674] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11124.474700] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11124.480140] workqueue events_power_efficient: flags=0x80 [11124.482637] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11124.485923] pending: neigh_periodic_work, do_cache_clean [11124.488560] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11124.492610] pending: neigh_periodic_work, check_lifetime [11124.495732] workqueue mm_percpu_wq: flags=0x8 [11124.498633] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11124.502178] pending: vmstat_update [11124.504410] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11124.508396] pending: vmstat_update [11124.510430] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11124.514369] pending: vmstat_update [11124.516976] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11124.521092] pending: vmstat_update [11124.523109] workqueue writeback: flags=0x4a [11124.525542] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11124.528901] pending: wb_workfn, wb_workfn [11124.531403] workqueue xprtiod: flags=0xa [11124.533592] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11124.537161] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11124.541408] workqueue ipv6_addrconf: flags=0x40008 [11124.545238] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11124.559153] pending: addrconf_verify_work [11124.564009] workqueue knbd0-recv: flags=0x1a [11124.567920] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11124.572181] in-flight: 647:recv_work [11124.576660] workqueue ptlrpc_pinger: flags=0x2 [11124.578605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11124.582574] pending: ptlrpc_pinger_main [ptlrpc] [11124.585785] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11114s workers=2 idle: 652 [11128.056048] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11128.058161] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11128.071733] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11128.075463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11128.078447] RIP: 0010:native_safe_halt+0x2e/0x50 [11128.080103] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11128.086880] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11128.089311] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11128.091595] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11128.093931] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11128.096312] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11128.098428] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11128.100644] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11128.103199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11128.104571] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11128.106861] Call Trace: [11128.107656] [11128.108267] ? show_regs.cold.9+0x22/0x2f [11128.109649] ? watchdog_timer_fn.cold.6+0x62/0x10a [11128.111132] ? __hrtimer_run_queues+0x1c2/0x4d0 [11128.112377] ? watchdog+0x40/0x40 [11128.113392] ? hrtimer_interrupt+0x128/0x2c0 [11128.114946] ? smp_apic_timer_interrupt+0xa6/0x270 [11128.116339] ? apic_timer_interrupt+0xf/0x20 [11128.117614] [11128.118205] ? native_safe_halt+0x2e/0x50 [11128.119280] kvm_wait+0xcd/0xe0 [11128.120167] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11128.121823] do_raw_spin_lock+0x177/0x190 [11128.123094] _raw_spin_lock+0x21/0x30 [11128.124263] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11128.126272] ? do_raw_spin_unlock+0x75/0x190 [11128.127411] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11128.129161] ? do_wait_intr+0xf0/0xf0 [11128.130158] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11128.131892] mdc_close+0x53b/0xd70 [mdc] [11128.133177] lmv_close+0x18d/0x490 [lmv] [11128.134298] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11128.135961] ll_md_real_close+0xa5/0x2a0 [lustre] [11128.137484] ll_md_close+0x274/0x800 [lustre] [11128.138738] ll_file_release+0x128/0xc50 [lustre] [11128.140311] __fput+0x10d/0x440 [11128.141345] ____fput+0x12/0x20 [11128.142253] task_work_run+0xc6/0x110 [11128.143348] exit_to_usermode_loop+0x1dd/0x1f0 [11128.144524] do_syscall_64+0x3d6/0x440 [11128.145625] entry_SYSCALL_64_after_hwframe+0x49/0xae [11128.147332] RIP: 0033:0x7f1a63983cd8 [11128.148637] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11128.153739] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11128.155769] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11128.157845] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11128.160127] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11128.161963] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11128.163957] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11136.058784] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [11136.065088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11136.096078] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11136.101644] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11136.106027] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11136.111611] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11136.120640] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11136.124109] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11136.127512] RDX: 0000000000007487 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11136.131837] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11136.136491] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11136.139311] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11136.142170] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11136.145749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11136.147790] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11136.150810] Call Trace: [11136.152344] [11136.153211] ? show_regs.cold.9+0x22/0x2f [11136.155251] ? watchdog_timer_fn.cold.6+0x62/0x10a [11136.157852] ? __hrtimer_run_queues+0x1c2/0x4d0 [11136.160721] ? watchdog+0x40/0x40 [11136.162392] ? hrtimer_interrupt+0x128/0x2c0 [11136.164222] ? smp_apic_timer_interrupt+0xa6/0x270 [11136.166215] ? apic_timer_interrupt+0xf/0x20 [11136.167894] [11136.168844] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11136.170607] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11136.172842] do_raw_spin_lock+0x177/0x190 [11136.174513] _raw_spin_lock+0x21/0x30 [11136.176007] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11136.178102] obd_mod_free+0x46/0x140 [ptlrpc] [11136.180118] mdc_commit_open+0x91/0xb0 [mdc] [11136.181533] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11136.183691] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11136.186036] after_reply+0xe2c/0x1780 [ptlrpc] [11136.188102] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11136.190230] ? __next_timer_interrupt+0x160/0x160 [11136.191752] ptlrpcd+0xa82/0xe40 [ptlrpc] [11136.193343] ? do_wait_intr+0xf0/0xf0 [11136.194640] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11136.197149] kthread+0x1d1/0x200 [11136.198247] ? set_kthread_struct+0x70/0x70 [11136.199424] ret_from_fork+0x1f/0x30 [11152.053125] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [11152.057096] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11152.060097] Modules linked in: [11152.066013] Modules linked in: [11152.066016] lustre(O) osp(O) [11152.067901] lustre(O) [11152.069199] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11152.069284] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11152.069290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11152.069297] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11152.069314] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11152.069321] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11152.069326] RAX: 00000000000078d4 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11152.069329] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11152.069331] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11152.069333] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11152.069336] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11152.069339] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11152.069342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11152.069344] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11152.069356] Call Trace: [11152.069365] [11152.069373] ? show_regs.cold.9+0x22/0x2f [11152.069386] ? watchdog_timer_fn.cold.6+0x62/0x10a [11152.069392] ? __hrtimer_run_queues+0x1c2/0x4d0 [11152.069400] ? watchdog+0x40/0x40 [11152.069408] ? hrtimer_interrupt+0x128/0x2c0 [11152.069412] ? smp_apic_timer_interrupt+0xa6/0x270 [11152.069426] ? apic_timer_interrupt+0xf/0x20 [11152.069431] [11152.069434] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11152.069441] do_raw_spin_lock+0x177/0x190 [11152.069446] _raw_spin_lock+0x21/0x30 [11152.069462] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11152.069769] ? do_raw_spin_unlock+0x75/0x190 [11152.069779] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11152.072835] osp(O) [11152.073725] ? do_wait_intr+0xf0/0xf0 [11152.090719] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11152.090910] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11152.090921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11152.090925] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11152.090940] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11152.090944] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11152.090949] RAX: 0000000000003014 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11152.090951] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11152.090954] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11152.090956] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11152.090958] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11152.090962] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11152.090965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11152.090968] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11152.090975] Call Trace: [11152.090980] [11152.090988] ? show_regs.cold.9+0x22/0x2f [11152.090996] ? watchdog_timer_fn.cold.6+0x62/0x10a [11152.091003] ? __hrtimer_run_queues+0x1c2/0x4d0 [11152.091007] ? watchdog+0x40/0x40 [11152.091010] ? hrtimer_interrupt+0x128/0x2c0 [11152.091014] ? smp_apic_timer_interrupt+0xa6/0x270 [11152.091021] ? apic_timer_interrupt+0xf/0x20 [11152.091025] [11152.091027] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11152.091033] do_raw_spin_lock+0x177/0x190 [11152.091036] _raw_spin_lock+0x21/0x30 [11152.091040] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11152.091374] ? do_raw_spin_unlock+0x75/0x190 [11152.091380] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11152.091617] ? do_wait_intr+0xf0/0xf0 [11152.091625] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11152.091949] mdc_close+0x53b/0xd70 [mdc] [11152.091987] lmv_close+0x18d/0x490 [lmv] [11152.092016] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11152.092145] ll_md_real_close+0xa5/0x2a0 [lustre] [11152.092231] ll_md_close+0x274/0x800 [lustre] [11152.092323] ll_file_release+0x128/0xc50 [lustre] [11152.092413] __fput+0x10d/0x440 [11152.092422] ____fput+0x12/0x20 [11152.092431] task_work_run+0xc6/0x110 [11152.092439] exit_to_usermode_loop+0x1dd/0x1f0 [11152.092502] do_syscall_64+0x3d6/0x440 [11152.092507] entry_SYSCALL_64_after_hwframe+0x49/0xae [11152.092512] RIP: 0033:0x7f1a63983cd8 [11152.092518] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11152.092522] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11152.092526] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11152.092529] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11152.092531] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11152.092533] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11152.092536] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11152.344372] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11152.347520] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11152.350672] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11152.355286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11152.361355] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11152.363535] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11152.367217] mdc_intent_lock+0x353/0x7e0 [mdc] [11152.371038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11152.374395] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11152.378408] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11152.382340] lmv_intent_open+0x338/0x1150 [lmv] [11152.384395] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11152.387162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11152.390054] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11152.392491] ll_intent_lock+0xc5/0xaa0 [lustre] [11152.396748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11152.399682] ll_lookup_it+0xe0e/0x2270 [lustre] [11152.401708] ? do_raw_spin_unlock+0x75/0x190 [11152.404044] ? _raw_spin_unlock+0x12/0x30 [11152.406686] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11152.409622] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11152.412176] ll_atomic_open+0x36a/0xd70 [lustre] [11152.415286] lookup_open+0x356/0xb40 [11152.417573] do_last+0x313/0xfc0 [11152.419573] ? nd_jump_root+0xe5/0x160 [11152.421228] ? path_init+0x437/0x520 [11152.423032] path_openat+0xf7/0x500 [11152.426332] ? do_raw_spin_unlock+0x75/0x190 [11152.432241] do_filp_open+0x99/0x140 [11152.435320] ? getname_flags+0x6e/0x330 [11152.437776] ? __check_object_size+0xff/0x256 [11152.440849] ? do_raw_spin_unlock+0x75/0x190 [11152.443113] ? _raw_spin_unlock+0x12/0x30 [11152.445289] do_sys_openat2+0x2b4/0x410 [11152.447080] do_sys_open+0x73/0xa0 [11152.450132] __x64_sys_openat+0x24/0x30 [11152.453361] do_syscall_64+0xc1/0x440 [11152.455275] entry_SYSCALL_64_after_hwframe+0x49/0xae [11152.457804] RIP: 0033:0x7f1a63983332 [11152.459834] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11152.468903] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11152.473465] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11152.477243] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11152.480542] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11152.483742] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11152.487201] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11154.923956] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9131s! [11154.932979] Showing busy workqueues and worker pools: [11154.936714] workqueue events: flags=0x0 [11154.939089] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11154.942833] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11154.947262] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11154.951292] pending: sec_gc_main [ptlrpc] [11154.954375] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11154.961908] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11154.969560] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11154.976463] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11154.982418] workqueue events_long: flags=0x0 [11154.989529] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11154.994027] pending: lu_objects_flush [obdclass] [11154.997358] workqueue events_unbound: flags=0x2 [11155.000246] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11155.004193] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11155.015317] workqueue events_power_efficient: flags=0x80 [11155.017382] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11155.027407] pending: neigh_periodic_work, do_cache_clean [11155.032032] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11155.036359] pending: neigh_periodic_work, check_lifetime [11155.039929] workqueue mm_percpu_wq: flags=0x8 [11155.043700] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11155.047769] pending: vmstat_update [11155.050240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11155.056784] pending: vmstat_update [11155.059610] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11155.064117] pending: vmstat_update [11155.067091] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11155.072711] pending: vmstat_update [11155.077076] workqueue writeback: flags=0x4a [11155.082223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11155.086685] pending: wb_workfn, wb_workfn [11155.092691] workqueue xprtiod: flags=0xa [11155.094711] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11155.102910] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11155.108855] workqueue ipv6_addrconf: flags=0x40008 [11155.113036] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11155.119683] pending: addrconf_verify_work [11155.124907] workqueue knbd0-recv: flags=0x1a [11155.131393] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11155.144410] in-flight: 647:recv_work [11155.151936] workqueue ptlrpc_pinger: flags=0x2 [11155.156019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11155.164214] pending: ptlrpc_pinger_main [ptlrpc] [11155.169344] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11145s workers=2 idle: 652 [11156.056028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11156.060817] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11156.110781] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11156.125747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11156.139211] RIP: 0010:native_safe_halt+0x2e/0x50 [11156.142362] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11156.165468] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11156.170298] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11156.177951] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11156.182675] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11156.187827] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11156.195257] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11156.200926] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11156.209427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11156.216468] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11156.223981] Call Trace: [11156.225996] [11156.228279] ? show_regs.cold.9+0x22/0x2f [11156.232585] ? watchdog_timer_fn.cold.6+0x62/0x10a [11156.237040] ? __hrtimer_run_queues+0x1c2/0x4d0 [11156.239263] ? watchdog+0x40/0x40 [11156.241854] ? hrtimer_interrupt+0x128/0x2c0 [11156.245087] ? smp_apic_timer_interrupt+0xa6/0x270 [11156.246692] ? apic_timer_interrupt+0xf/0x20 [11156.255867] [11156.258566] ? native_safe_halt+0x2e/0x50 [11156.262271] kvm_wait+0xcd/0xe0 [11156.263588] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11156.271560] do_raw_spin_lock+0x177/0x190 [11156.275864] _raw_spin_lock+0x21/0x30 [11156.277781] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11156.284098] ? do_raw_spin_unlock+0x75/0x190 [11156.289083] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11156.292474] ? do_wait_intr+0xf0/0xf0 [11156.299741] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11156.310474] mdc_close+0x53b/0xd70 [mdc] [11156.316203] lmv_close+0x18d/0x490 [lmv] [11156.319625] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11156.326552] ll_md_real_close+0xa5/0x2a0 [lustre] [11156.333314] ll_md_close+0x274/0x800 [lustre] [11156.337739] ll_file_release+0x128/0xc50 [lustre] [11156.340361] __fput+0x10d/0x440 [11156.343475] ____fput+0x12/0x20 [11156.345524] task_work_run+0xc6/0x110 [11156.352048] exit_to_usermode_loop+0x1dd/0x1f0 [11156.362856] do_syscall_64+0x3d6/0x440 [11156.367778] entry_SYSCALL_64_after_hwframe+0x49/0xae [11156.370381] RIP: 0033:0x7f1a63983cd8 [11156.372200] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11156.389455] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11156.398548] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11156.402896] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11156.407107] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11156.411933] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11156.419892] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11164.059186] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [11164.067506] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11164.108533] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11164.122551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11164.129574] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11164.134822] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11164.153097] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11164.159246] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11164.164237] RDX: 0000000000001b4c RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11164.170055] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11164.177455] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11164.182570] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11164.187547] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11164.192413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11164.196268] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11164.200437] Call Trace: [11164.202067] [11164.203326] ? show_regs.cold.9+0x22/0x2f [11164.205420] ? watchdog_timer_fn.cold.6+0x62/0x10a [11164.211025] ? __hrtimer_run_queues+0x1c2/0x4d0 [11164.215539] ? watchdog+0x40/0x40 [11164.218426] ? hrtimer_interrupt+0x128/0x2c0 [11164.222312] ? smp_apic_timer_interrupt+0xa6/0x270 [11164.224522] ? apic_timer_interrupt+0xf/0x20 [11164.227567] [11164.229127] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11164.231516] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11164.238737] do_raw_spin_lock+0x177/0x190 [11164.243072] _raw_spin_lock+0x21/0x30 [11164.246123] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11164.249795] obd_mod_free+0x46/0x140 [ptlrpc] [11164.253150] mdc_commit_open+0x91/0xb0 [mdc] [11164.255544] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11164.264350] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11164.267991] after_reply+0xe2c/0x1780 [ptlrpc] [11164.273446] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11164.279323] ? __next_timer_interrupt+0x160/0x160 [11164.285137] ptlrpcd+0xa82/0xe40 [ptlrpc] [11164.290053] ? do_wait_intr+0xf0/0xf0 [11164.292585] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11164.295629] kthread+0x1d1/0x200 [11164.299279] ? set_kthread_struct+0x70/0x70 [11164.301514] ret_from_fork+0x1f/0x30 [11180.053064] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [11180.057017] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11180.062403] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [11180.065228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11180.065305] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11180.065312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11180.065315] RIP: 0010:native_safe_halt+0x2e/0x50 [11180.065329] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11180.065333] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11180.066089] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11180.066092] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [11180.066094] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11180.066096] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11180.066098] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11180.066102] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11180.066104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11180.066107] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11180.066114] Call Trace: [11180.066119] [11180.066125] ? show_regs.cold.9+0x22/0x2f [11180.066135] ? watchdog_timer_fn.cold.6+0x62/0x10a [11180.066141] ? __hrtimer_run_queues+0x1c2/0x4d0 [11180.066145] ? watchdog+0x40/0x40 [11180.066149] ? hrtimer_interrupt+0x128/0x2c0 [11180.066153] ? smp_apic_timer_interrupt+0xa6/0x270 [11180.066159] ? apic_timer_interrupt+0xf/0x20 [11180.066163] [11180.066166] ? native_safe_halt+0x2e/0x50 [11180.066170] kvm_wait+0xcd/0xe0 [11180.066176] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11180.066183] do_raw_spin_lock+0x177/0x190 [11180.066187] _raw_spin_lock+0x21/0x30 [11180.066191] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11180.067152] ? do_raw_spin_unlock+0x75/0x190 [11180.067159] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11180.072407] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11180.072470] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11180.072478] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11180.072481] RIP: 0010:native_safe_halt+0x2e/0x50 [11180.072507] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11180.072512] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11180.072516] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11180.072519] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [11180.072521] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11180.072523] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11180.072525] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11180.072528] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11180.072531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11180.072534] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11180.072543] Call Trace: [11180.072548] [11180.072554] ? show_regs.cold.9+0x22/0x2f [11180.072566] ? watchdog_timer_fn.cold.6+0x62/0x10a [11180.072573] ? __hrtimer_run_queues+0x1c2/0x4d0 [11180.072578] ? watchdog+0x40/0x40 [11180.072583] ? hrtimer_interrupt+0x128/0x2c0 [11180.072662] ? smp_apic_timer_interrupt+0xa6/0x270 [11180.072670] ? apic_timer_interrupt+0xf/0x20 [11180.072674] [11180.072678] ? native_safe_halt+0x2e/0x50 [11180.072682] kvm_wait+0xcd/0xe0 [11180.072691] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11180.072698] do_raw_spin_lock+0x177/0x190 [11180.072704] _raw_spin_lock+0x21/0x30 [11180.072709] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11180.092866] ? do_wait_intr+0xf0/0xf0 [11180.092995] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11180.093257] mdc_close+0x53b/0xd70 [mdc] [11180.093294] lmv_close+0x18d/0x490 [lmv] [11180.093325] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11180.093613] ll_md_real_close+0xa5/0x2a0 [lustre] [11180.093692] ll_md_close+0x274/0x800 [lustre] [11180.093773] ll_file_release+0x128/0xc50 [lustre] [11180.093853] __fput+0x10d/0x440 [11180.093862] ____fput+0x12/0x20 [11180.093867] task_work_run+0xc6/0x110 [11180.094011] exit_to_usermode_loop+0x1dd/0x1f0 [11180.094018] do_syscall_64+0x3d6/0x440 [11180.094023] entry_SYSCALL_64_after_hwframe+0x49/0xae [11180.094031] RIP: 0033:0x7f1a63983cd8 [11180.094040] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11180.094044] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11180.094049] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11180.094051] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11180.094053] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11180.094055] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11180.094057] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11180.338513] ? do_raw_spin_unlock+0x75/0x190 [11180.339907] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11180.341852] ? do_wait_intr+0xf0/0xf0 [11180.342917] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11180.344844] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11180.346617] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11180.348609] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11180.350469] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11180.352004] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11180.353580] mdc_intent_lock+0x353/0x7e0 [mdc] [11180.355323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11180.356883] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11180.359829] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11180.364426] lmv_intent_open+0x338/0x1150 [lmv] [11180.365876] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11180.369003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11180.372472] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11180.376842] ll_intent_lock+0xc5/0xaa0 [lustre] [11180.378102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11180.380753] ll_lookup_it+0xe0e/0x2270 [lustre] [11180.382429] ? do_raw_spin_unlock+0x75/0x190 [11180.383927] ? _raw_spin_unlock+0x12/0x30 [11180.386854] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11180.391014] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11180.393423] ll_atomic_open+0x36a/0xd70 [lustre] [11180.395331] lookup_open+0x356/0xb40 [11180.397791] do_last+0x313/0xfc0 [11180.399355] ? nd_jump_root+0xe5/0x160 [11180.400344] ? path_init+0x437/0x520 [11180.402397] path_openat+0xf7/0x500 [11180.404413] ? do_raw_spin_unlock+0x75/0x190 [11180.405892] do_filp_open+0x99/0x140 [11180.406855] ? getname_flags+0x6e/0x330 [11180.408802] ? __check_object_size+0xff/0x256 [11180.410014] ? do_raw_spin_unlock+0x75/0x190 [11180.411286] ? _raw_spin_unlock+0x12/0x30 [11180.413503] do_sys_openat2+0x2b4/0x410 [11180.415298] do_sys_open+0x73/0xa0 [11180.416453] __x64_sys_openat+0x24/0x30 [11180.420395] do_syscall_64+0xc1/0x440 [11180.421451] entry_SYSCALL_64_after_hwframe+0x49/0xae [11180.422826] RIP: 0033:0x7f1a63983332 [11180.424096] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11180.433252] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11180.442856] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11180.444887] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11180.448836] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11180.451254] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11180.453120] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11184.056593] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11184.060223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11184.082427] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11184.086523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11184.090215] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11184.092646] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11184.100573] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11184.103143] RAX: 000000000000512d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11184.105747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11184.108744] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11184.111222] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11184.113426] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11184.116244] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11184.118864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11184.120802] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11184.123026] Call Trace: [11184.123850] [11184.124538] ? show_regs.cold.9+0x22/0x2f [11184.125732] ? watchdog_timer_fn.cold.6+0x62/0x10a [11184.127564] ? __hrtimer_run_queues+0x1c2/0x4d0 [11184.129060] ? watchdog+0x40/0x40 [11184.130079] ? hrtimer_interrupt+0x128/0x2c0 [11184.131418] ? smp_apic_timer_interrupt+0xa6/0x270 [11184.133194] ? apic_timer_interrupt+0xf/0x20 [11184.134756] [11184.135386] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11184.137236] do_raw_spin_lock+0x177/0x190 [11184.139259] _raw_spin_lock+0x21/0x30 [11184.140640] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11184.142649] ? do_raw_spin_unlock+0x75/0x190 [11184.144404] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11184.146711] ? do_wait_intr+0xf0/0xf0 [11184.148410] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11184.151751] mdc_close+0x53b/0xd70 [mdc] [11184.153814] lmv_close+0x18d/0x490 [lmv] [11184.156016] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11184.159197] ll_md_real_close+0xa5/0x2a0 [lustre] [11184.161677] ll_md_close+0x274/0x800 [lustre] [11184.163745] ll_file_release+0x128/0xc50 [lustre] [11184.165634] __fput+0x10d/0x440 [11184.167150] ____fput+0x12/0x20 [11184.168378] task_work_run+0xc6/0x110 [11184.169837] exit_to_usermode_loop+0x1dd/0x1f0 [11184.171341] do_syscall_64+0x3d6/0x440 [11184.172827] entry_SYSCALL_64_after_hwframe+0x49/0xae [11184.174772] RIP: 0033:0x7f1a63983cd8 [11184.176059] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11184.182577] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11184.185742] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11184.188828] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11184.192105] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11184.195200] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11184.198338] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11185.632231] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9162s! [11185.638119] Showing busy workqueues and worker pools: [11185.641129] workqueue events: flags=0x0 [11185.643572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11185.646693] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11185.650984] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11185.657867] pending: sec_gc_main [ptlrpc] [11185.661636] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11185.664919] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11185.672041] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11185.675301] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11185.680914] workqueue events_long: flags=0x0 [11185.683408] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11185.687881] pending: lu_objects_flush [obdclass] [11185.690981] workqueue events_unbound: flags=0x2 [11185.693141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11185.696574] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11185.701976] workqueue events_power_efficient: flags=0x80 [11185.705216] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11185.709245] pending: neigh_periodic_work, do_cache_clean [11185.711985] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11185.715350] pending: neigh_periodic_work, check_lifetime [11185.718062] workqueue mm_percpu_wq: flags=0x8 [11185.720616] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11185.723845] pending: vmstat_update [11185.725759] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11185.729188] pending: vmstat_update [11185.731375] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11185.735056] pending: vmstat_update [11185.736759] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11185.740347] pending: vmstat_update [11185.742344] workqueue writeback: flags=0x4a [11185.744900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11185.748301] pending: wb_workfn, wb_workfn [11185.750572] workqueue xprtiod: flags=0xa [11185.753171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11185.756109] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11185.760694] workqueue ipv6_addrconf: flags=0x40008 [11185.762847] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11185.767291] pending: addrconf_verify_work [11185.770375] workqueue knbd0-recv: flags=0x1a [11185.772206] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11185.777077] in-flight: 647:recv_work [11185.779463] workqueue ptlrpc_pinger: flags=0x2 [11185.781994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11185.785278] pending: ptlrpc_pinger_main [ptlrpc] [11185.793770] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11175s workers=2 idle: 652 [11192.058610] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11192.062119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11192.082927] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11192.090929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11192.095502] RIP: 0010:native_safe_halt+0x2e/0x50 [11192.098351] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11192.108892] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11192.113724] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11192.117861] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11192.122098] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11192.125689] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11192.129248] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11192.132244] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11192.137066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11192.140731] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11192.144594] Call Trace: [11192.146333] [11192.148982] ? show_regs.cold.9+0x22/0x2f [11192.151884] ? watchdog_timer_fn.cold.6+0x62/0x10a [11192.154375] ? __hrtimer_run_queues+0x1c2/0x4d0 [11192.159266] ? watchdog+0x40/0x40 [11192.160509] ? hrtimer_interrupt+0x128/0x2c0 [11192.164578] ? smp_apic_timer_interrupt+0xa6/0x270 [11192.166199] ? apic_timer_interrupt+0xf/0x20 [11192.167692] [11192.168540] ? native_safe_halt+0x2e/0x50 [11192.170142] kvm_wait+0xcd/0xe0 [11192.171197] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11192.173507] do_raw_spin_lock+0x177/0x190 [11192.175066] _raw_spin_lock+0x21/0x30 [11192.176525] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11192.178623] obd_mod_free+0x46/0x140 [ptlrpc] [11192.180558] mdc_commit_open+0x91/0xb0 [mdc] [11192.183907] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11192.188925] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11192.191888] after_reply+0xe2c/0x1780 [ptlrpc] [11192.196795] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11192.202185] ? __next_timer_interrupt+0x160/0x160 [11192.205641] ptlrpcd+0xa82/0xe40 [ptlrpc] [11192.207723] ? do_wait_intr+0xf0/0xf0 [11192.208812] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11192.211948] kthread+0x1d1/0x200 [11192.213601] ? set_kthread_struct+0x70/0x70 [11192.215641] ret_from_fork+0x1f/0x30 [11208.057289] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11208.060599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11208.090376] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11208.097470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11208.107396] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11208.111640] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11208.125779] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11208.128924] RAX: 0000000000004e5f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11208.133725] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11208.136729] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11208.139773] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11208.149473] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11208.153446] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11208.158671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11208.163146] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11208.166482] Call Trace: [11208.169743] [11208.170768] ? show_regs.cold.9+0x22/0x2f [11208.174523] ? watchdog_timer_fn.cold.6+0x62/0x10a [11208.177502] ? __hrtimer_run_queues+0x1c2/0x4d0 [11208.179664] ? watchdog+0x40/0x40 [11208.181369] ? hrtimer_interrupt+0x128/0x2c0 [11208.188478] ? smp_apic_timer_interrupt+0xa6/0x270 [11208.194082] ? apic_timer_interrupt+0xf/0x20 [11208.197832] [11208.198552] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11208.200103] do_raw_spin_lock+0x177/0x190 [11208.206513] _raw_spin_lock+0x21/0x30 [11208.207728] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11208.211797] ? do_raw_spin_unlock+0x75/0x190 [11208.213713] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11208.215812] ? do_wait_intr+0xf0/0xf0 [11208.220281] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11208.222548] mdc_close+0x53b/0xd70 [mdc] [11208.223682] lmv_close+0x18d/0x490 [lmv] [11208.225754] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11208.228591] ll_md_real_close+0xa5/0x2a0 [lustre] [11208.230806] ll_md_close+0x274/0x800 [lustre] [11208.233777] ll_file_release+0x128/0xc50 [lustre] [11208.243419] __fput+0x10d/0x440 [11208.247117] ____fput+0x12/0x20 [11208.248608] task_work_run+0xc6/0x110 [11208.255342] exit_to_usermode_loop+0x1dd/0x1f0 [11208.258763] do_syscall_64+0x3d6/0x440 [11208.263279] entry_SYSCALL_64_after_hwframe+0x49/0xae [11208.268111] RIP: 0033:0x7f1a63983cd8 [11208.271137] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11208.292655] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11208.297197] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11208.311276] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11208.317276] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11208.320776] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11208.325477] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11212.056122] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [11212.059843] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [11212.059848] Modules linked in: lustre(O) [11212.069723] Modules linked in: lustre(O) [11212.072025] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11212.072120] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11212.072128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11212.072130] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11212.072143] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11212.072147] RSP: 0018:ffffc38542073af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11212.076946] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11212.077026] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11212.077033] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11212.077036] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11212.077050] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11212.077053] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11212.077058] RAX: 0000000000005b8c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11212.077060] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11212.077063] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11212.077065] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11212.077067] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11212.077070] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11212.077073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11212.077075] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11212.077083] Call Trace: [11212.077088] [11212.077094] ? show_regs.cold.9+0x22/0x2f [11212.077106] ? watchdog_timer_fn.cold.6+0x62/0x10a [11212.077113] ? __hrtimer_run_queues+0x1c2/0x4d0 [11212.077117] ? watchdog+0x40/0x40 [11212.077121] ? hrtimer_interrupt+0x128/0x2c0 [11212.077126] ? smp_apic_timer_interrupt+0xa6/0x270 [11212.077135] ? apic_timer_interrupt+0xf/0x20 [11212.077141] [11212.077144] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11212.099935] RAX: 0000000000006611 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11212.099949] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11212.099952] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11212.099954] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11212.099956] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11212.100136] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11212.100140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11212.100147] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11212.100157] Call Trace: [11212.100173] [11212.100186] ? show_regs.cold.9+0x22/0x2f [11212.100198] ? watchdog_timer_fn.cold.6+0x62/0x10a [11212.100204] ? __hrtimer_run_queues+0x1c2/0x4d0 [11212.100207] ? watchdog+0x40/0x40 [11212.100212] ? hrtimer_interrupt+0x128/0x2c0 [11212.100216] ? smp_apic_timer_interrupt+0xa6/0x270 [11212.100224] ? apic_timer_interrupt+0xf/0x20 [11212.100229] [11212.100232] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11212.100241] do_raw_spin_lock+0x177/0x190 [11212.100246] _raw_spin_lock+0x21/0x30 [11212.100257] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11212.100546] ? do_raw_spin_unlock+0x75/0x190 [11212.100552] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11212.100902] ? do_wait_intr+0xf0/0xf0 [11212.100910] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11212.101327] mdc_close+0x53b/0xd70 [mdc] [11212.101367] lmv_close+0x18d/0x490 [lmv] [11212.101398] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11212.101514] ll_md_real_close+0xa5/0x2a0 [lustre] [11212.101675] ll_md_close+0x274/0x800 [lustre] [11212.101768] ll_file_release+0x128/0xc50 [lustre] [11212.101862] __fput+0x10d/0x440 [11212.101870] ____fput+0x12/0x20 [11212.101879] task_work_run+0xc6/0x110 [11212.101887] exit_to_usermode_loop+0x1dd/0x1f0 [11212.101894] do_syscall_64+0x3d6/0x440 [11212.101903] entry_SYSCALL_64_after_hwframe+0x49/0xae [11212.101914] RIP: 0033:0x7f1a63983cd8 [11212.101922] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11212.101926] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11212.101931] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11212.101933] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11212.101935] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11212.101938] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11212.101939] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11212.537928] do_raw_spin_lock+0x177/0x190 [11212.539947] _raw_spin_lock+0x21/0x30 [11212.544586] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11212.554763] ? do_raw_spin_unlock+0x75/0x190 [11212.561587] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11212.563977] ? do_wait_intr+0xf0/0xf0 [11212.565591] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11212.568483] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11212.571659] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11212.574964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11212.582998] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11212.588546] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11212.593338] mdc_intent_lock+0x353/0x7e0 [mdc] [11212.596652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11212.602932] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11212.609336] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11212.612405] lmv_intent_open+0x338/0x1150 [lmv] [11212.614017] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11212.617561] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11212.623812] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11212.625714] ll_intent_lock+0xc5/0xaa0 [lustre] [11212.632268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11212.634843] ll_lookup_it+0xe0e/0x2270 [lustre] [11212.637047] ? do_raw_spin_unlock+0x75/0x190 [11212.642109] ? _raw_spin_unlock+0x12/0x30 [11212.643071] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11212.644688] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11212.646708] ll_atomic_open+0x36a/0xd70 [lustre] [11212.648515] lookup_open+0x356/0xb40 [11212.649711] do_last+0x313/0xfc0 [11212.651550] ? nd_jump_root+0xe5/0x160 [11212.653043] ? path_init+0x437/0x520 [11212.654714] path_openat+0xf7/0x500 [11212.656217] ? do_raw_spin_unlock+0x75/0x190 [11212.657695] do_filp_open+0x99/0x140 [11212.658872] ? getname_flags+0x6e/0x330 [11212.659929] ? __check_object_size+0xff/0x256 [11212.661047] ? do_raw_spin_unlock+0x75/0x190 [11212.662735] ? _raw_spin_unlock+0x12/0x30 [11212.663688] do_sys_openat2+0x2b4/0x410 [11212.664629] do_sys_open+0x73/0xa0 [11212.666019] __x64_sys_openat+0x24/0x30 [11212.672486] do_syscall_64+0xc1/0x440 [11212.674724] entry_SYSCALL_64_after_hwframe+0x49/0xae [11212.683155] RIP: 0033:0x7f1a63983332 [11212.688591] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11212.718506] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11212.726541] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11212.739144] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11212.745036] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11212.750940] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11212.758653] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11215.840361] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9192s! [11215.848096] Showing busy workqueues and worker pools: [11215.861767] workqueue events: flags=0x0 [11215.869837] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11215.886206] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11215.889115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11215.910072] pending: sec_gc_main [ptlrpc] [11215.922959] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11215.934324] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11215.948383] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11215.954645] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11215.966846] workqueue events_long: flags=0x0 [11215.969018] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11215.978350] pending: lu_objects_flush [obdclass] [11215.989017] workqueue events_unbound: flags=0x2 [11215.998478] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11216.012157] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11216.031560] workqueue events_power_efficient: flags=0x80 [11216.037349] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11216.045259] pending: neigh_periodic_work, do_cache_clean [11216.055111] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11216.069167] pending: neigh_periodic_work, check_lifetime [11216.078336] workqueue mm_percpu_wq: flags=0x8 [11216.084352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11216.095388] pending: vmstat_update [11216.098333] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11216.111328] pending: vmstat_update [11216.116331] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11216.123925] pending: vmstat_update [11216.125982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11216.133584] pending: vmstat_update [11216.136433] workqueue writeback: flags=0x4a [11216.144230] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11216.153698] pending: wb_workfn, wb_workfn [11216.155813] workqueue xprtiod: flags=0xa [11216.168118] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11216.170351] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11216.179221] workqueue ipv6_addrconf: flags=0x40008 [11216.181727] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11216.189094] pending: addrconf_verify_work [11216.191363] workqueue knbd0-recv: flags=0x1a [11216.195272] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11216.203107] in-flight: 647:recv_work [11216.210508] workqueue ptlrpc_pinger: flags=0x2 [11216.220152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11216.227600] pending: ptlrpc_pinger_main [ptlrpc] [11216.242387] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11206s workers=2 idle: 652 [11220.058020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11220.067518] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11220.093487] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11220.102668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11220.107823] RIP: 0010:native_safe_halt+0x2e/0x50 [11220.110845] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11220.122837] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11220.128564] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11220.133457] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11220.137880] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11220.142546] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11220.147525] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11220.151424] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11220.155847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11220.158775] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11220.177313] Call Trace: [11220.181806] [11220.182590] ? show_regs.cold.9+0x22/0x2f [11220.190924] ? watchdog_timer_fn.cold.6+0x62/0x10a [11220.197295] ? __hrtimer_run_queues+0x1c2/0x4d0 [11220.202452] ? watchdog+0x40/0x40 [11220.205261] ? hrtimer_interrupt+0x128/0x2c0 [11220.214407] ? smp_apic_timer_interrupt+0xa6/0x270 [11220.223814] ? apic_timer_interrupt+0xf/0x20 [11220.225896] [11220.231022] ? native_safe_halt+0x2e/0x50 [11220.236326] kvm_wait+0xcd/0xe0 [11220.240694] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11220.247909] do_raw_spin_lock+0x177/0x190 [11220.252212] _raw_spin_lock+0x21/0x30 [11220.258861] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11220.271266] obd_mod_free+0x46/0x140 [ptlrpc] [11220.274440] mdc_commit_open+0x91/0xb0 [mdc] [11220.278451] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11220.283216] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11220.290010] after_reply+0xe2c/0x1780 [ptlrpc] [11220.294954] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11220.299275] ? __next_timer_interrupt+0x160/0x160 [11220.301899] ptlrpcd+0xa82/0xe40 [ptlrpc] [11220.304481] ? do_wait_intr+0xf0/0xf0 [11220.307275] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11220.310879] kthread+0x1d1/0x200 [11220.312852] ? set_kthread_struct+0x70/0x70 [11220.315766] ret_from_fork+0x1f/0x30 [11236.057636] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11236.060556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11236.102160] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11236.116089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11236.126928] RIP: 0010:native_safe_halt+0x2e/0x50 [11236.131608] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11236.153339] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11236.159459] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11236.167630] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [11236.173583] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11236.179046] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11236.182236] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11236.185544] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11236.191857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11236.195590] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11236.204768] Call Trace: [11236.207146] [11236.207958] ? show_regs.cold.9+0x22/0x2f [11236.212652] ? watchdog_timer_fn.cold.6+0x62/0x10a [11236.216680] ? __hrtimer_run_queues+0x1c2/0x4d0 [11236.219203] ? watchdog+0x40/0x40 [11236.220458] ? hrtimer_interrupt+0x128/0x2c0 [11236.224101] ? smp_apic_timer_interrupt+0xa6/0x270 [11236.227672] ? apic_timer_interrupt+0xf/0x20 [11236.229822] [11236.231711] ? native_safe_halt+0x2e/0x50 [11236.235511] kvm_wait+0xcd/0xe0 [11236.236748] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11236.240394] do_raw_spin_lock+0x177/0x190 [11236.241840] _raw_spin_lock+0x21/0x30 [11236.246916] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11236.250480] ? do_raw_spin_unlock+0x75/0x190 [11236.252498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11236.255853] ? do_wait_intr+0xf0/0xf0 [11236.262218] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11236.265394] mdc_close+0x53b/0xd70 [mdc] [11236.271920] lmv_close+0x18d/0x490 [lmv] [11236.275087] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11236.277935] ll_md_real_close+0xa5/0x2a0 [lustre] [11236.288378] ll_md_close+0x274/0x800 [lustre] [11236.289612] ll_file_release+0x128/0xc50 [lustre] [11236.294263] __fput+0x10d/0x440 [11236.299275] ____fput+0x12/0x20 [11236.303670] task_work_run+0xc6/0x110 [11236.306343] exit_to_usermode_loop+0x1dd/0x1f0 [11236.311862] do_syscall_64+0x3d6/0x440 [11236.315235] entry_SYSCALL_64_after_hwframe+0x49/0xae [11236.316665] RIP: 0033:0x7f1a63983cd8 [11236.319733] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11236.339123] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11236.344206] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11236.354352] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11236.360166] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11236.366182] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11236.371740] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11240.058483] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [11240.061492] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11240.079270] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11240.083383] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11240.086429] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11240.088424] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11240.093854] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11240.096093] RAX: 000000000000374f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11240.098318] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11240.100580] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11240.102953] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11240.105480] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11240.107739] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11240.110316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11240.111976] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11240.113973] Call Trace: [11240.114795] [11240.115584] ? show_regs.cold.9+0x22/0x2f [11240.117214] ? watchdog_timer_fn.cold.6+0x62/0x10a [11240.118882] ? __hrtimer_run_queues+0x1c2/0x4d0 [11240.120349] ? watchdog+0x40/0x40 [11240.121328] ? hrtimer_interrupt+0x128/0x2c0 [11240.122498] ? smp_apic_timer_interrupt+0xa6/0x270 [11240.124236] ? apic_timer_interrupt+0xf/0x20 [11240.125719] [11240.126483] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11240.128309] do_raw_spin_lock+0x177/0x190 [11240.129525] _raw_spin_lock+0x21/0x30 [11240.130614] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11240.132666] ? do_raw_spin_unlock+0x75/0x190 [11240.134041] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11240.136392] ? do_wait_intr+0xf0/0xf0 [11240.137590] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11240.139638] mdc_close+0x53b/0xd70 [mdc] [11240.141186] lmv_close+0x18d/0x490 [lmv] [11240.142360] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11240.144243] ll_md_real_close+0xa5/0x2a0 [lustre] [11240.145889] ll_md_close+0x274/0x800 [lustre] [11240.147448] ll_file_release+0x128/0xc50 [lustre] [11240.149296] __fput+0x10d/0x440 [11240.150322] ____fput+0x12/0x20 [11240.151358] task_work_run+0xc6/0x110 [11240.152556] exit_to_usermode_loop+0x1dd/0x1f0 [11240.154007] do_syscall_64+0x3d6/0x440 [11240.155299] entry_SYSCALL_64_after_hwframe+0x49/0xae [11240.156703] RIP: 0033:0x7f1a63983cd8 [11240.157870] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11240.163570] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11240.165647] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11240.168022] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11240.170334] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11240.172460] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11240.174690] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11244.055308] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11244.061718] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11244.110655] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11244.119646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11244.130771] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11244.138720] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11244.157803] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11244.165583] RAX: 0000000000006683 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11244.175685] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11244.184596] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11244.192772] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11244.199798] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11244.203602] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11244.208711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11244.217373] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11244.224383] Call Trace: [11244.226570] [11244.228826] ? show_regs.cold.9+0x22/0x2f [11244.231981] ? watchdog_timer_fn.cold.6+0x62/0x10a [11244.237591] ? __hrtimer_run_queues+0x1c2/0x4d0 [11244.242319] ? watchdog+0x40/0x40 [11244.246724] ? hrtimer_interrupt+0x128/0x2c0 [11244.250367] ? smp_apic_timer_interrupt+0xa6/0x270 [11244.254827] ? apic_timer_interrupt+0xf/0x20 [11244.256267] [11244.257580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11244.259607] do_raw_spin_lock+0x177/0x190 [11244.261547] _raw_spin_lock+0x21/0x30 [11244.263529] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11244.266941] ? do_raw_spin_unlock+0x75/0x190 [11244.269350] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11244.272154] ? do_wait_intr+0xf0/0xf0 [11244.273531] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11244.276360] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11244.280754] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11244.283182] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11244.286212] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11244.288389] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11244.290945] mdc_intent_lock+0x353/0x7e0 [mdc] [11244.293788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11244.297126] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11244.300522] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11244.303663] lmv_intent_open+0x338/0x1150 [lmv] [11244.305987] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11244.309133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11244.312815] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11244.315419] ll_intent_lock+0xc5/0xaa0 [lustre] [11244.318146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11244.321419] ll_lookup_it+0xe0e/0x2270 [lustre] [11244.323647] ? do_raw_spin_unlock+0x75/0x190 [11244.326449] ? _raw_spin_unlock+0x12/0x30 [11244.328616] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11244.331052] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11244.333723] ll_atomic_open+0x36a/0xd70 [lustre] [11244.340138] lookup_open+0x356/0xb40 [11244.346229] do_last+0x313/0xfc0 [11244.349578] ? nd_jump_root+0xe5/0x160 [11244.353744] ? path_init+0x437/0x520 [11244.357036] path_openat+0xf7/0x500 [11244.361573] ? do_raw_spin_unlock+0x75/0x190 [11244.366803] do_filp_open+0x99/0x140 [11244.368944] ? getname_flags+0x6e/0x330 [11244.374638] ? __check_object_size+0xff/0x256 [11244.380341] ? do_raw_spin_unlock+0x75/0x190 [11244.383383] ? _raw_spin_unlock+0x12/0x30 [11244.388968] do_sys_openat2+0x2b4/0x410 [11244.392789] do_sys_open+0x73/0xa0 [11244.397558] __x64_sys_openat+0x24/0x30 [11244.402766] do_syscall_64+0xc1/0x440 [11244.407426] entry_SYSCALL_64_after_hwframe+0x49/0xae [11244.412543] RIP: 0033:0x7f1a63983332 [11244.416982] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11244.438555] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11244.447124] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11244.452686] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11244.459725] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11244.480730] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11244.492061] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11246.560138] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9223s! [11246.566168] Showing busy workqueues and worker pools: [11246.569386] workqueue events: flags=0x0 [11246.571524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11246.577909] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11246.582234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11246.588887] pending: sec_gc_main [ptlrpc] [11246.592829] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11246.596620] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11246.619137] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11246.630379] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11246.644615] workqueue events_long: flags=0x0 [11246.648552] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11246.651951] pending: lu_objects_flush [obdclass] [11246.658766] workqueue events_unbound: flags=0x2 [11246.662596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11246.666185] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11246.673374] workqueue events_power_efficient: flags=0x80 [11246.677651] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11246.681564] pending: neigh_periodic_work, do_cache_clean [11246.692139] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11246.698404] pending: neigh_periodic_work, check_lifetime [11246.702573] workqueue mm_percpu_wq: flags=0x8 [11246.706488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11246.715735] pending: vmstat_update [11246.722427] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11246.725444] pending: vmstat_update [11246.731759] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11246.738888] pending: vmstat_update [11246.742641] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11246.751611] pending: vmstat_update [11246.754502] workqueue writeback: flags=0x4a [11246.758727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11246.765466] pending: wb_workfn, wb_workfn [11246.768365] workqueue xprtiod: flags=0xa [11246.770520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11246.775298] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11246.783799] workqueue ipv6_addrconf: flags=0x40008 [11246.789419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11246.795092] pending: addrconf_verify_work [11246.799645] workqueue knbd0-recv: flags=0x1a [11246.803545] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11246.810714] in-flight: 647:recv_work [11246.815627] workqueue ptlrpc_pinger: flags=0x2 [11246.824995] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11246.833477] pending: ptlrpc_pinger_main [ptlrpc] [11246.838591] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11236s workers=2 idle: 652 [11248.060694] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11248.068328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11248.135549] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11248.143774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11248.149968] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11248.152888] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11248.160798] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11248.174263] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11248.181848] RDX: 0000000000000cf8 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11248.196617] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11248.205440] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11248.208103] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11248.212766] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11248.232325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11248.244988] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11248.247554] Call Trace: [11248.260561] [11248.262960] ? show_regs.cold.9+0x22/0x2f [11248.268649] ? watchdog_timer_fn.cold.6+0x62/0x10a [11248.270415] ? __hrtimer_run_queues+0x1c2/0x4d0 [11248.278123] ? watchdog+0x40/0x40 [11248.289027] ? hrtimer_interrupt+0x128/0x2c0 [11248.294486] ? smp_apic_timer_interrupt+0xa6/0x270 [11248.300003] ? apic_timer_interrupt+0xf/0x20 [11248.306471] [11248.308314] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11248.312475] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11248.315790] do_raw_spin_lock+0x177/0x190 [11248.317206] _raw_spin_lock+0x21/0x30 [11248.319460] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11248.330701] obd_mod_free+0x46/0x140 [ptlrpc] [11248.332638] mdc_commit_open+0x91/0xb0 [mdc] [11248.336244] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11248.340162] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11248.344296] after_reply+0xe2c/0x1780 [ptlrpc] [11248.349869] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11248.353132] ? __next_timer_interrupt+0x160/0x160 [11248.355782] ptlrpcd+0xa82/0xe40 [ptlrpc] [11248.358440] ? do_wait_intr+0xf0/0xf0 [11248.361867] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11248.374309] kthread+0x1d1/0x200 [11248.377893] ? set_kthread_struct+0x70/0x70 [11248.382533] ret_from_fork+0x1f/0x30 [11262.653299] rcu: INFO: rcu_sched self-detected stall on CPU [11262.656658] rcu: 3-...!: (5828921 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [11262.670825] (t=9240264 jiffies g=269633 q=3272) [11262.675846] rcu: rcu_sched kthread starved for 9239915 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [11262.690790] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11262.702698] rcu: RCU grace-period kthread stack dump: [11262.711571] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11262.721274] Call Trace: [11262.723231] __schedule+0x351/0xcb0 [11262.726589] schedule+0xc0/0x180 [11262.729647] schedule_timeout+0xb4/0x190 [11262.730676] ? do_raw_spin_unlock+0x75/0x190 [11262.734479] ? __next_timer_interrupt+0x160/0x160 [11262.737524] ? prepare_to_swait_event+0xf3/0x140 [11262.740537] rcu_gp_kthread+0x8dd/0x15b0 [11262.741581] ? force_qs_rnp+0x2c0/0x2c0 [11262.742613] kthread+0x1d1/0x200 [11262.751664] ? set_kthread_struct+0x70/0x70 [11262.755098] ret_from_fork+0x1f/0x30 [11262.759062] rcu: Stack dump where RCU GP kthread last ran: [11262.763447] Sending NMI from CPU 3 to CPUs 0: [11262.770423] NMI backtrace for cpu 0 [11262.770430] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11262.770432] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11262.770434] RIP: 0010:interrupt_entry+0xa0/0xb9 [11262.770436] Code: 45 31 ed 45 31 f6 45 31 ff f6 84 24 90 00 00 00 03 74 0a e9 05 00 00 00 e8 8d 36 00 ff 48 8d 7c 24 08 65 ff 04 25 cc dc 01 00 <75> 12 65 48 89 3c 25 f8 5f 00 00 65 48 8b 24 25 d0 dc 01 00 57 ff [11262.770438] RSP: 0018:ffffc38541c37440 EFLAGS: 00000056 [11262.770441] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000 [11262.770443] RDX: 0000000000000000 RSI: ffffffff9020188a RDI: ffffc38541c37448 [11262.770445] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [11262.770446] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [11262.770448] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11262.770449] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11262.770451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11262.770453] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11262.770454] Call Trace: [11262.770455] RIP: 0010:native_safe_halt+0x2e/0x50 [11262.770458] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11262.770460] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11262.770463] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11262.770465] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [11262.770467] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11262.770468] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11262.770470] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11262.770472] ? kvm_wait+0xcd/0xe0 [11262.770473] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11262.770474] ? do_raw_spin_lock+0x177/0x190 [11262.770475] ? _raw_spin_lock+0x21/0x30 [11262.770476] ? ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11262.770478] ? do_raw_spin_unlock+0x75/0x190 [11262.770479] ? ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11262.770480] ? do_wait_intr+0xf0/0xf0 [11262.770481] ? ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11262.770483] ? ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11262.770484] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11262.770485] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11262.770487] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11262.770488] ? mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11262.770490] ? mdc_intent_lock+0x353/0x7e0 [mdc] [11262.770491] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11262.770492] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11262.770494] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11262.770495] ? lmv_intent_open+0x338/0x1150 [lmv] [11262.770496] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11262.770498] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11262.770499] ? lmv_intent_lock+0x1dd/0x4c0 [lmv] [11262.770500] ? ll_intent_lock+0xc5/0xaa0 [lustre] [11262.770501] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11262.770503] ? ll_lookup_it+0xe0e/0x2270 [lustre] [11262.770504] ? do_raw_spin_unlock+0x75/0x190 [11262.770505] ? _raw_spin_unlock+0x12/0x30 [11262.770507] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11262.770508] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11262.770509] ? ll_atomic_open+0x36a/0xd70 [lustre] [11262.770511] ? lookup_open+0x356/0xb40 [11262.770512] ? do_last+0x313/0xfc0 [11262.770513] ? nd_jump_root+0xe5/0x160 [11262.770514] ? path_init+0x437/0x520 [11262.770515] ? path_openat+0xf7/0x500 [11262.770516] ? do_raw_spin_unlock+0x75/0x190 [11262.770517] ? do_filp_open+0x99/0x140 [11262.770519] ? getname_flags+0x6e/0x330 [11262.770520] ? __check_object_size+0xff/0x256 [11262.770521] ? do_raw_spin_unlock+0x75/0x190 [11262.770522] ? _raw_spin_unlock+0x12/0x30 [11262.770524] ? do_sys_openat2+0x2b4/0x410 [11262.770525] ? do_sys_open+0x73/0xa0 [11262.770526] ? __x64_sys_openat+0x24/0x30 [11262.770527] ? do_syscall_64+0xc1/0x440 [11262.770528] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [11262.771030] NMI backtrace for cpu 3 [11263.218131] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11263.229966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11263.235619] Call Trace: [11263.237379] [11263.238604] ? dump_stack+0xbb/0x10e [11263.244100] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11263.249616] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11263.253126] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11263.259969] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11263.264604] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11263.273099] ? rcu_dump_cpu_stacks+0x11b/0x171 [11263.279979] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11263.283931] ? update_process_times+0x6d/0xc0 [11263.286566] ? tick_sched_handle.isra.18+0x27/0x90 [11263.290934] ? tick_sched_timer+0x53/0xb0 [11263.293207] ? __hrtimer_run_queues+0x1c2/0x4d0 [11263.296380] ? tick_sched_handle.isra.18+0x90/0x90 [11263.302412] ? hrtimer_interrupt+0x128/0x2c0 [11263.305095] ? smp_apic_timer_interrupt+0xa6/0x270 [11263.308628] ? apic_timer_interrupt+0xf/0x20 [11263.311464] [11263.312806] ? native_safe_halt+0x2e/0x50 [11263.315065] ? kvm_wait+0xcd/0xe0 [11263.318386] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11263.322282] ? do_raw_spin_lock+0x177/0x190 [11263.326079] ? _raw_spin_lock+0x21/0x30 [11263.327532] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11263.333451] ? obd_mod_free+0x46/0x140 [ptlrpc] [11263.342043] ? mdc_commit_open+0x91/0xb0 [mdc] [11263.347145] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11263.352202] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11263.357352] ? after_reply+0xe2c/0x1780 [ptlrpc] [11263.362203] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11263.366423] ? __next_timer_interrupt+0x160/0x160 [11263.369531] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11263.372807] ? do_wait_intr+0xf0/0xf0 [11263.375212] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11263.379519] ? kthread+0x1d1/0x200 [11263.381594] ? set_kthread_struct+0x70/0x70 [11263.383975] ? ret_from_fork+0x1f/0x30 [11264.057149] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11264.062276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11264.085648] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11264.091351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11264.097972] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11264.100651] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11264.109280] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11264.112856] RAX: 0000000000004892 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11264.116207] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11264.119807] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11264.123136] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11264.126483] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11264.130198] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11264.134203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11264.137269] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11264.140803] Call Trace: [11264.142325] [11264.143632] ? show_regs.cold.9+0x22/0x2f [11264.145438] ? watchdog_timer_fn.cold.6+0x62/0x10a [11264.147473] ? __hrtimer_run_queues+0x1c2/0x4d0 [11264.151372] ? watchdog+0x40/0x40 [11264.153637] ? hrtimer_interrupt+0x128/0x2c0 [11264.157796] ? smp_apic_timer_interrupt+0xa6/0x270 [11264.162135] ? apic_timer_interrupt+0xf/0x20 [11264.167127] [11264.168006] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11264.174345] do_raw_spin_lock+0x177/0x190 [11264.176058] _raw_spin_lock+0x21/0x30 [11264.181505] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11264.184281] ? do_raw_spin_unlock+0x75/0x190 [11264.186151] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11264.187978] ? do_wait_intr+0xf0/0xf0 [11264.191261] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11264.195627] mdc_close+0x53b/0xd70 [mdc] [11264.197219] lmv_close+0x18d/0x490 [lmv] [11264.200517] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11264.211061] ll_md_real_close+0xa5/0x2a0 [lustre] [11264.212568] ll_md_close+0x274/0x800 [lustre] [11264.213872] ll_file_release+0x128/0xc50 [lustre] [11264.220759] __fput+0x10d/0x440 [11264.223763] ____fput+0x12/0x20 [11264.225096] task_work_run+0xc6/0x110 [11264.227116] exit_to_usermode_loop+0x1dd/0x1f0 [11264.230383] do_syscall_64+0x3d6/0x440 [11264.232487] entry_SYSCALL_64_after_hwframe+0x49/0xae [11264.235393] RIP: 0033:0x7f1a63983cd8 [11264.237603] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11264.250812] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11264.255374] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11264.259488] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11264.263078] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11264.266831] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11264.270493] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11268.058350] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [11268.067982] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11268.113320] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11268.128097] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11268.145578] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11268.157928] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11268.182598] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11268.189042] RAX: 0000000000003609 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11268.195355] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11268.201601] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11268.205335] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11268.211710] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11268.215249] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11268.226334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11268.233675] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11268.239930] Call Trace: [11268.249067] [11268.253896] ? show_regs.cold.9+0x22/0x2f [11268.260227] ? watchdog_timer_fn.cold.6+0x62/0x10a [11268.267370] ? __hrtimer_run_queues+0x1c2/0x4d0 [11268.271568] ? watchdog+0x40/0x40 [11268.275283] ? hrtimer_interrupt+0x128/0x2c0 [11268.278301] ? smp_apic_timer_interrupt+0xa6/0x270 [11268.283258] ? apic_timer_interrupt+0xf/0x20 [11268.285217] [11268.288854] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11268.292768] do_raw_spin_lock+0x177/0x190 [11268.295280] _raw_spin_lock+0x21/0x30 [11268.302570] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11268.311703] ? do_raw_spin_unlock+0x75/0x190 [11268.316809] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11268.323605] ? do_wait_intr+0xf0/0xf0 [11268.328254] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11268.334927] mdc_close+0x53b/0xd70 [mdc] [11268.339062] lmv_close+0x18d/0x490 [lmv] [11268.340386] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11268.347316] ll_md_real_close+0xa5/0x2a0 [lustre] [11268.352313] ll_md_close+0x274/0x800 [lustre] [11268.357134] ll_file_release+0x128/0xc50 [lustre] [11268.362933] __fput+0x10d/0x440 [11268.365407] ____fput+0x12/0x20 [11268.367195] task_work_run+0xc6/0x110 [11268.371083] exit_to_usermode_loop+0x1dd/0x1f0 [11268.378569] do_syscall_64+0x3d6/0x440 [11268.383592] entry_SYSCALL_64_after_hwframe+0x49/0xae [11268.386839] RIP: 0033:0x7f1a63983cd8 [11268.393572] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11268.404506] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11268.411943] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11268.417583] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11268.426516] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11268.433765] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11268.438396] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11277.281020] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9254s! [11277.290577] Showing busy workqueues and worker pools: [11277.294188] workqueue events: flags=0x0 [11277.302010] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11277.314962] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11277.322177] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11277.332310] pending: sec_gc_main [ptlrpc] [11277.337672] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11277.343475] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11277.354384] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11277.357528] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11277.368347] workqueue events_long: flags=0x0 [11277.378112] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11277.394978] pending: lu_objects_flush [obdclass] [11277.403202] workqueue events_unbound: flags=0x2 [11277.413614] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11277.417699] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11277.437351] workqueue events_power_efficient: flags=0x80 [11277.444414] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11277.455206] pending: neigh_periodic_work, do_cache_clean [11277.466051] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11277.472003] pending: neigh_periodic_work, check_lifetime [11277.474109] workqueue mm_percpu_wq: flags=0x8 [11277.480181] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11277.486388] pending: vmstat_update [11277.490431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11277.497107] pending: vmstat_update [11277.499886] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11277.506562] pending: vmstat_update [11277.522822] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11277.525051] pending: vmstat_update [11277.537707] workqueue writeback: flags=0x4a [11277.542166] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11277.547804] pending: wb_workfn, wb_workfn [11277.551013] workqueue xprtiod: flags=0xa [11277.554494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11277.556827] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11277.559514] workqueue ipv6_addrconf: flags=0x40008 [11277.561364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11277.567246] pending: addrconf_verify_work [11277.568823] workqueue knbd0-recv: flags=0x1a [11277.576256] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11277.580000] in-flight: 647:recv_work [11277.586051] workqueue ptlrpc_pinger: flags=0x2 [11277.589159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11277.593609] pending: ptlrpc_pinger_main [ptlrpc] [11277.596657] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11267s workers=2 idle: 652 [11288.058276] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11288.067826] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11288.111859] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11288.127993] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11288.136042] RIP: 0010:native_safe_halt+0x2e/0x50 [11288.139866] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11288.159002] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11288.167021] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11288.181073] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11288.193198] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11288.200653] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11288.208114] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11288.219747] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11288.229276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11288.238032] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11288.242295] Call Trace: [11288.243782] [11288.244771] ? show_regs.cold.9+0x22/0x2f [11288.253148] ? watchdog_timer_fn.cold.6+0x62/0x10a [11288.260160] ? __hrtimer_run_queues+0x1c2/0x4d0 [11288.265914] ? watchdog+0x40/0x40 [11288.269269] ? hrtimer_interrupt+0x128/0x2c0 [11288.272745] ? smp_apic_timer_interrupt+0xa6/0x270 [11288.278458] ? apic_timer_interrupt+0xf/0x20 [11288.281983] [11288.284602] ? native_safe_halt+0x2e/0x50 [11288.287325] kvm_wait+0xcd/0xe0 [11288.288968] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11288.291218] do_raw_spin_lock+0x177/0x190 [11288.292611] _raw_spin_lock+0x21/0x30 [11288.300197] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11288.303570] obd_mod_free+0x46/0x140 [ptlrpc] [11288.308223] mdc_commit_open+0x91/0xb0 [mdc] [11288.309813] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11288.320121] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11288.325875] after_reply+0xe2c/0x1780 [ptlrpc] [11288.331101] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11288.336260] ? __next_timer_interrupt+0x160/0x160 [11288.339981] ptlrpcd+0xa82/0xe40 [ptlrpc] [11288.344652] ? do_wait_intr+0xf0/0xf0 [11288.349619] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11288.353060] kthread+0x1d1/0x200 [11288.359104] ? set_kthread_struct+0x70/0x70 [11288.364913] ret_from_fork+0x1f/0x30 [11292.057039] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11292.061598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11292.116237] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11292.127238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11292.141167] RIP: 0010:native_safe_halt+0x2e/0x50 [11292.143602] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11292.156533] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11292.161035] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11292.164870] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [11292.168713] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11292.173150] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11292.179165] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11292.184829] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11292.189532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11292.193317] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11292.198880] Call Trace: [11292.199911] [11292.201415] ? show_regs.cold.9+0x22/0x2f [11292.203668] ? watchdog_timer_fn.cold.6+0x62/0x10a [11292.205995] ? __hrtimer_run_queues+0x1c2/0x4d0 [11292.210032] ? watchdog+0x40/0x40 [11292.211797] ? hrtimer_interrupt+0x128/0x2c0 [11292.215426] ? smp_apic_timer_interrupt+0xa6/0x270 [11292.220987] ? apic_timer_interrupt+0xf/0x20 [11292.229799] [11292.233890] ? native_safe_halt+0x2e/0x50 [11292.239629] kvm_wait+0xcd/0xe0 [11292.249690] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11292.262774] do_raw_spin_lock+0x177/0x190 [11292.267792] _raw_spin_lock+0x21/0x30 [11292.273818] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11292.275957] ? do_raw_spin_unlock+0x75/0x190 [11292.292870] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11292.302696] ? do_wait_intr+0xf0/0xf0 [11292.305244] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11292.320910] mdc_close+0x53b/0xd70 [mdc] [11292.326825] lmv_close+0x18d/0x490 [lmv] [11292.334916] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11292.339565] ll_md_real_close+0xa5/0x2a0 [lustre] [11292.347311] ll_md_close+0x274/0x800 [lustre] [11292.350486] ll_file_release+0x128/0xc50 [lustre] [11292.352744] __fput+0x10d/0x440 [11292.354089] ____fput+0x12/0x20 [11292.355530] task_work_run+0xc6/0x110 [11292.357208] exit_to_usermode_loop+0x1dd/0x1f0 [11292.359672] do_syscall_64+0x3d6/0x440 [11292.361671] entry_SYSCALL_64_after_hwframe+0x49/0xae [11292.364259] RIP: 0033:0x7f1a63983cd8 [11292.366552] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11292.383109] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11292.391894] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11292.401518] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11292.405857] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11292.410084] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11292.426761] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11296.059250] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [11296.071810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11296.119123] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11296.128580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11296.139789] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11296.147110] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11296.163321] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11296.169619] RAX: 0000000000003ef7 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11296.175603] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11296.181120] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11296.189469] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11296.195499] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11296.199607] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11296.208956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11296.217527] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11296.222944] Call Trace: [11296.226734] [11296.227415] ? show_regs.cold.9+0x22/0x2f [11296.232764] ? watchdog_timer_fn.cold.6+0x62/0x10a [11296.235735] ? __hrtimer_run_queues+0x1c2/0x4d0 [11296.240187] ? watchdog+0x40/0x40 [11296.243053] ? hrtimer_interrupt+0x128/0x2c0 [11296.246279] ? smp_apic_timer_interrupt+0xa6/0x270 [11296.248141] ? apic_timer_interrupt+0xf/0x20 [11296.254772] [11296.255817] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11296.259709] do_raw_spin_lock+0x177/0x190 [11296.263183] _raw_spin_lock+0x21/0x30 [11296.266780] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11296.271476] ? do_raw_spin_unlock+0x75/0x190 [11296.276944] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11296.278929] ? do_wait_intr+0xf0/0xf0 [11296.282040] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11296.287675] mdc_close+0x53b/0xd70 [mdc] [11296.289811] lmv_close+0x18d/0x490 [lmv] [11296.291955] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11296.297741] ll_md_real_close+0xa5/0x2a0 [lustre] [11296.299858] ll_md_close+0x274/0x800 [lustre] [11296.302220] ll_file_release+0x128/0xc50 [lustre] [11296.307878] __fput+0x10d/0x440 [11296.309052] ____fput+0x12/0x20 [11296.310246] task_work_run+0xc6/0x110 [11296.318811] exit_to_usermode_loop+0x1dd/0x1f0 [11296.322822] do_syscall_64+0x3d6/0x440 [11296.324926] entry_SYSCALL_64_after_hwframe+0x49/0xae [11296.338840] RIP: 0033:0x7f1a63983cd8 [11296.341210] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11296.355348] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11296.365797] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11296.374322] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11296.383853] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11296.388577] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11296.393167] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11304.053031] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11304.058592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11304.098159] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11304.104209] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11304.113731] RIP: 0010:native_safe_halt+0x2e/0x50 [11304.116044] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11304.137070] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11304.141122] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11304.144888] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [11304.150020] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11304.156905] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11304.161416] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11304.164977] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11304.168341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11304.171087] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11304.174566] Call Trace: [11304.176389] [11304.177615] ? show_regs.cold.9+0x22/0x2f [11304.179510] ? watchdog_timer_fn.cold.6+0x62/0x10a [11304.182255] ? __hrtimer_run_queues+0x1c2/0x4d0 [11304.187099] ? watchdog+0x40/0x40 [11304.192131] ? hrtimer_interrupt+0x128/0x2c0 [11304.194622] ? smp_apic_timer_interrupt+0xa6/0x270 [11304.200191] ? apic_timer_interrupt+0xf/0x20 [11304.205539] [11304.206406] ? native_safe_halt+0x2e/0x50 [11304.212795] kvm_wait+0xcd/0xe0 [11304.216158] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11304.222117] do_raw_spin_lock+0x177/0x190 [11304.225164] _raw_spin_lock+0x21/0x30 [11304.228425] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11304.234843] ? do_raw_spin_unlock+0x75/0x190 [11304.236526] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11304.244574] ? do_wait_intr+0xf0/0xf0 [11304.248048] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11304.251207] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11304.258672] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11304.265338] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11304.269866] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11304.275603] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11304.279990] mdc_intent_lock+0x353/0x7e0 [mdc] [11304.282524] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11304.285598] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11304.289185] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11304.295568] lmv_intent_open+0x338/0x1150 [lmv] [11304.298428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11304.302131] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11304.305661] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11304.307967] ll_intent_lock+0xc5/0xaa0 [lustre] [11304.314018] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11304.322636] ll_lookup_it+0xe0e/0x2270 [lustre] [11304.331655] ? do_raw_spin_unlock+0x75/0x190 [11304.341825] ? _raw_spin_unlock+0x12/0x30 [11304.350609] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11304.358661] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11304.367791] ll_atomic_open+0x36a/0xd70 [lustre] [11304.371962] lookup_open+0x356/0xb40 [11304.377403] do_last+0x313/0xfc0 [11304.380095] ? nd_jump_root+0xe5/0x160 [11304.389057] ? path_init+0x437/0x520 [11304.392580] path_openat+0xf7/0x500 [11304.397166] ? do_raw_spin_unlock+0x75/0x190 [11304.399057] do_filp_open+0x99/0x140 [11304.400231] ? getname_flags+0x6e/0x330 [11304.404331] ? __check_object_size+0xff/0x256 [11304.412676] ? do_raw_spin_unlock+0x75/0x190 [11304.419063] ? _raw_spin_unlock+0x12/0x30 [11304.427843] do_sys_openat2+0x2b4/0x410 [11304.433866] do_sys_open+0x73/0xa0 [11304.434942] __x64_sys_openat+0x24/0x30 [11304.437271] do_syscall_64+0xc1/0x440 [11304.438415] entry_SYSCALL_64_after_hwframe+0x49/0xae [11304.441482] RIP: 0033:0x7f1a63983332 [11304.443194] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11304.457245] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11304.464405] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11304.471124] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11304.476222] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11304.478511] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11304.485981] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11308.000037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9284s! [11308.004353] Showing busy workqueues and worker pools: [11308.007113] workqueue events: flags=0x0 [11308.009351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11308.013122] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11308.017332] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11308.020338] pending: sec_gc_main [ptlrpc] [11308.022695] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11308.025426] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11308.029553] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11308.032388] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11308.036163] workqueue events_long: flags=0x0 [11308.037728] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11308.040612] pending: lu_objects_flush [obdclass] [11308.042766] workqueue events_unbound: flags=0x2 [11308.044169] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11308.046453] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11308.049898] workqueue events_power_efficient: flags=0x80 [11308.051762] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11308.054104] pending: neigh_periodic_work, do_cache_clean [11308.056071] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11308.058404] pending: neigh_periodic_work, check_lifetime [11308.060686] workqueue mm_percpu_wq: flags=0x8 [11308.062744] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11308.065165] pending: vmstat_update [11308.066454] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11308.068731] pending: vmstat_update [11308.070009] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11308.072447] pending: vmstat_update [11308.073504] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11308.075642] pending: vmstat_update [11308.076888] workqueue writeback: flags=0x4a [11308.078156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11308.080050] pending: wb_workfn, wb_workfn [11308.081472] workqueue xprtiod: flags=0xa [11308.082704] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11308.084679] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11308.086977] workqueue ipv6_addrconf: flags=0x40008 [11308.088578] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11308.090523] pending: addrconf_verify_work [11308.091859] workqueue knbd0-recv: flags=0x1a [11308.093371] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11308.095361] in-flight: 647:recv_work [11308.096628] workqueue ptlrpc_pinger: flags=0x2 [11308.097986] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11308.099894] pending: ptlrpc_pinger_main [ptlrpc] [11308.101926] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11298s workers=2 idle: 652 [11316.059572] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11316.063889] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11316.112729] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11316.124876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11316.132129] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11316.136734] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11316.155584] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11316.159836] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11316.170057] RDX: 0000000000003f44 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11316.178486] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11316.180986] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11316.186138] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11316.198700] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11316.209973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11316.213588] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11316.222063] Call Trace: [11316.223093] [11316.226010] ? show_regs.cold.9+0x22/0x2f [11316.228281] ? watchdog_timer_fn.cold.6+0x62/0x10a [11316.232140] ? __hrtimer_run_queues+0x1c2/0x4d0 [11316.237015] ? watchdog+0x40/0x40 [11316.240547] ? hrtimer_interrupt+0x128/0x2c0 [11316.243640] ? smp_apic_timer_interrupt+0xa6/0x270 [11316.245932] ? apic_timer_interrupt+0xf/0x20 [11316.248107] [11316.251832] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11316.255237] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11316.262471] do_raw_spin_lock+0x177/0x190 [11316.263876] _raw_spin_lock+0x21/0x30 [11316.274005] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11316.276002] obd_mod_free+0x46/0x140 [ptlrpc] [11316.280711] mdc_commit_open+0x91/0xb0 [mdc] [11316.286313] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11316.290032] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11316.294460] after_reply+0xe2c/0x1780 [ptlrpc] [11316.299761] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11316.307489] ? __next_timer_interrupt+0x160/0x160 [11316.309855] ptlrpcd+0xa82/0xe40 [ptlrpc] [11316.317913] ? do_wait_intr+0xf0/0xf0 [11316.320987] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11316.326020] kthread+0x1d1/0x200 [11316.331232] ? set_kthread_struct+0x70/0x70 [11316.332674] ret_from_fork+0x1f/0x30 [11320.058442] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11320.069333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11320.162081] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11320.194452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11320.212393] RIP: 0010:native_safe_halt+0x2e/0x50 [11320.221683] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11320.261064] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11320.277352] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11320.283731] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [11320.297049] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11320.305006] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11320.321092] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11320.336571] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11320.346267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11320.356893] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11320.368063] Call Trace: [11320.369923] [11320.374543] ? show_regs.cold.9+0x22/0x2f [11320.384324] ? watchdog_timer_fn.cold.6+0x62/0x10a [11320.391103] ? __hrtimer_run_queues+0x1c2/0x4d0 [11320.397208] ? watchdog+0x40/0x40 [11320.399917] ? hrtimer_interrupt+0x128/0x2c0 [11320.401666] ? smp_apic_timer_interrupt+0xa6/0x270 [11320.408905] ? apic_timer_interrupt+0xf/0x20 [11320.413030] [11320.414504] ? native_safe_halt+0x2e/0x50 [11320.416860] kvm_wait+0xcd/0xe0 [11320.418684] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11320.425480] do_raw_spin_lock+0x177/0x190 [11320.432420] _raw_spin_lock+0x21/0x30 [11320.435308] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11320.443239] ? do_raw_spin_unlock+0x75/0x190 [11320.445429] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11320.448252] ? do_wait_intr+0xf0/0xf0 [11320.449843] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11320.451953] mdc_close+0x53b/0xd70 [mdc] [11320.453948] lmv_close+0x18d/0x490 [lmv] [11320.456015] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11320.460052] ll_md_real_close+0xa5/0x2a0 [lustre] [11320.462599] ll_md_close+0x274/0x800 [lustre] [11320.464787] ll_file_release+0x128/0xc50 [lustre] [11320.467258] __fput+0x10d/0x440 [11320.470187] ____fput+0x12/0x20 [11320.472719] task_work_run+0xc6/0x110 [11320.477679] exit_to_usermode_loop+0x1dd/0x1f0 [11320.480550] do_syscall_64+0x3d6/0x440 [11320.485838] entry_SYSCALL_64_after_hwframe+0x49/0xae [11320.490222] RIP: 0033:0x7f1a63983cd8 [11320.492069] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11320.505060] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11320.510777] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11320.516508] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11320.519850] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11320.523535] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11320.528683] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11324.057856] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11324.094004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11324.167674] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11324.180977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11324.186052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11324.189333] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11324.215275] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11324.222535] RAX: 00000000000033c4 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11324.237649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11324.239917] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11324.246309] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11324.255097] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11324.258881] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11324.270032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11324.280163] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11324.286272] Call Trace: [11324.287451] [11324.289382] ? show_regs.cold.9+0x22/0x2f [11324.293064] ? watchdog_timer_fn.cold.6+0x62/0x10a [11324.296849] ? __hrtimer_run_queues+0x1c2/0x4d0 [11324.299912] ? watchdog+0x40/0x40 [11324.301511] ? hrtimer_interrupt+0x128/0x2c0 [11324.303237] ? smp_apic_timer_interrupt+0xa6/0x270 [11324.305496] ? apic_timer_interrupt+0xf/0x20 [11324.309934] [11324.310528] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11324.315572] do_raw_spin_lock+0x177/0x190 [11324.322482] _raw_spin_lock+0x21/0x30 [11324.324496] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11324.333953] ? do_raw_spin_unlock+0x75/0x190 [11324.336115] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11324.340832] ? do_wait_intr+0xf0/0xf0 [11324.348864] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11324.350693] mdc_close+0x53b/0xd70 [mdc] [11324.360469] lmv_close+0x18d/0x490 [lmv] [11324.363222] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11324.372511] ll_md_real_close+0xa5/0x2a0 [lustre] [11324.381813] ll_md_close+0x274/0x800 [lustre] [11324.385970] ll_file_release+0x128/0xc50 [lustre] [11324.390477] __fput+0x10d/0x440 [11324.394742] ____fput+0x12/0x20 [11324.399387] task_work_run+0xc6/0x110 [11324.405403] exit_to_usermode_loop+0x1dd/0x1f0 [11324.411047] do_syscall_64+0x3d6/0x440 [11324.423525] entry_SYSCALL_64_after_hwframe+0x49/0xae [11324.430122] RIP: 0033:0x7f1a63983cd8 [11324.433945] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11324.461684] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11324.467576] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11324.470898] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11324.477619] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11324.493425] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11324.503609] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11336.053118] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11336.071941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11336.159395] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11336.190766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11336.201349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11336.215015] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11336.249802] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11336.258917] RAX: 00000000000075ac RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11336.278611] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11336.295378] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11336.310784] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11336.335022] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11336.347580] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11336.365731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11336.368073] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11336.375910] Call Trace: [11336.376880] [11336.394483] ? show_regs.cold.9+0x22/0x2f [11336.397533] ? watchdog_timer_fn.cold.6+0x62/0x10a [11336.402659] ? __hrtimer_run_queues+0x1c2/0x4d0 [11336.497406] ? watchdog+0x40/0x40 [11336.498465] ? hrtimer_interrupt+0x128/0x2c0 [11336.499721] ? smp_apic_timer_interrupt+0xa6/0x270 [11336.521093] ? apic_timer_interrupt+0xf/0x20 [11336.522309] [11336.522918] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11336.564705] do_raw_spin_lock+0x177/0x190 [11336.565913] _raw_spin_lock+0x21/0x30 [11336.573428] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11336.575184] ? do_raw_spin_unlock+0x75/0x190 [11336.612632] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11336.616812] ? do_wait_intr+0xf0/0xf0 [11336.617876] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11336.645546] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11336.664955] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11336.687754] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11336.689475] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11336.710850] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11336.712324] mdc_intent_lock+0x353/0x7e0 [mdc] [11336.731646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11336.733350] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11336.769075] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11336.780990] lmv_intent_open+0x338/0x1150 [lmv] [11336.787505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11336.802765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11336.812114] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11336.819551] ll_intent_lock+0xc5/0xaa0 [lustre] [11336.821091] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11336.839639] ll_lookup_it+0xe0e/0x2270 [lustre] [11336.844332] ? do_raw_spin_unlock+0x75/0x190 [11336.854361] ? _raw_spin_unlock+0x12/0x30 [11336.855562] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11336.874135] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11336.875685] ll_atomic_open+0x36a/0xd70 [lustre] [11336.889778] lookup_open+0x356/0xb40 [11336.915686] do_last+0x313/0xfc0 [11336.916685] ? nd_jump_root+0xe5/0x160 [11336.917803] ? path_init+0x437/0x520 [11336.918875] path_openat+0xf7/0x500 [11336.937392] ? do_raw_spin_unlock+0x75/0x190 [11336.949879] do_filp_open+0x99/0x140 [11336.952131] ? getname_flags+0x6e/0x330 [11336.963636] ? __check_object_size+0xff/0x256 [11336.972297] ? do_raw_spin_unlock+0x75/0x190 [11336.975214] ? _raw_spin_unlock+0x12/0x30 [11336.985635] do_sys_openat2+0x2b4/0x410 [11336.990805] do_sys_open+0x73/0xa0 [11337.003200] __x64_sys_openat+0x24/0x30 [11337.008636] do_syscall_64+0xc1/0x440 [11337.017023] entry_SYSCALL_64_after_hwframe+0x49/0xae [11337.020637] RIP: 0033:0x7f1a63983332 [11337.023525] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11337.053772] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11337.063757] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11337.072801] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11337.087562] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11337.099026] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11337.112065] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11338.212495] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9315s! [11338.223239] Showing busy workqueues and worker pools: [11338.224546] workqueue events: flags=0x0 [11338.244223] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11338.266672] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11338.268762] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11338.288819] pending: sec_gc_main [ptlrpc] [11338.300669] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11338.307525] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11338.319711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11338.326924] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11338.337672] workqueue events_long: flags=0x0 [11338.341529] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11338.346744] pending: lu_objects_flush [obdclass] [11338.369221] workqueue events_unbound: flags=0x2 [11338.370322] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11338.371771] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11338.387625] workqueue events_power_efficient: flags=0x80 [11338.394594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11338.409619] pending: neigh_periodic_work, do_cache_clean [11338.411300] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11338.423701] pending: neigh_periodic_work, check_lifetime [11338.425181] workqueue mm_percpu_wq: flags=0x8 [11338.436518] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11338.439851] pending: vmstat_update [11338.443192] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11338.454083] pending: vmstat_update [11338.463015] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11338.467210] pending: vmstat_update [11338.468614] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11338.478526] pending: vmstat_update [11338.482511] workqueue writeback: flags=0x4a [11338.488248] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11338.492327] pending: wb_workfn, wb_workfn [11338.501515] workqueue xprtiod: flags=0xa [11338.504206] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11338.511755] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11338.518774] workqueue ipv6_addrconf: flags=0x40008 [11338.523961] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11338.537668] pending: addrconf_verify_work [11338.540313] workqueue knbd0-recv: flags=0x1a [11338.547586] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11338.552273] in-flight: 647:recv_work [11338.553271] workqueue ptlrpc_pinger: flags=0x2 [11338.565903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11338.576332] pending: ptlrpc_pinger_main [ptlrpc] [11338.582464] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11328s workers=2 idle: 652 [11344.059385] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11344.073544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11344.136843] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11344.154405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11344.164589] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11344.176228] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11344.195587] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11344.202562] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11344.216795] RDX: 0000000000000d04 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11344.244183] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11344.257610] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11344.266544] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11344.276645] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11344.290989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11344.300246] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11344.310031] Call Trace: [11344.314575] [11344.315414] ? show_regs.cold.9+0x22/0x2f [11344.320161] ? watchdog_timer_fn.cold.6+0x62/0x10a [11344.331609] ? __hrtimer_run_queues+0x1c2/0x4d0 [11344.339614] ? watchdog+0x40/0x40 [11344.347294] ? hrtimer_interrupt+0x128/0x2c0 [11344.351614] ? smp_apic_timer_interrupt+0xa6/0x270 [11344.366524] ? apic_timer_interrupt+0xf/0x20 [11344.371996] [11344.380327] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11344.382298] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11344.395128] do_raw_spin_lock+0x177/0x190 [11344.401989] _raw_spin_lock+0x21/0x30 [11344.404058] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11344.409290] obd_mod_free+0x46/0x140 [ptlrpc] [11344.415690] mdc_commit_open+0x91/0xb0 [mdc] [11344.423692] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11344.431158] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11344.440749] after_reply+0xe2c/0x1780 [ptlrpc] [11344.444227] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11344.453450] ? __next_timer_interrupt+0x160/0x160 [11344.461108] ptlrpcd+0xa82/0xe40 [ptlrpc] [11344.468192] ? do_wait_intr+0xf0/0xf0 [11344.478819] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11344.490148] kthread+0x1d1/0x200 [11344.492667] ? set_kthread_struct+0x70/0x70 [11344.500105] ret_from_fork+0x1f/0x30 [11348.061412] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11348.077428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11348.175828] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11348.192270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11348.207404] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11348.219068] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11348.232840] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11348.236693] RAX: 0000000000002113 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11348.240130] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11348.251033] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11348.257427] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11348.267180] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11348.274869] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11348.292590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11348.295620] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11348.309897] Call Trace: [11348.313106] [11348.317677] ? show_regs.cold.9+0x22/0x2f [11348.327428] ? watchdog_timer_fn.cold.6+0x62/0x10a [11348.337571] ? __hrtimer_run_queues+0x1c2/0x4d0 [11348.340481] ? watchdog+0x40/0x40 [11348.347900] ? hrtimer_interrupt+0x128/0x2c0 [11348.353379] ? smp_apic_timer_interrupt+0xa6/0x270 [11348.360623] ? apic_timer_interrupt+0xf/0x20 [11348.366718] [11348.368593] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11348.376281] do_raw_spin_lock+0x177/0x190 [11348.378879] _raw_spin_lock+0x21/0x30 [11348.393345] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11348.402879] ? do_raw_spin_unlock+0x75/0x190 [11348.404949] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11348.411534] ? do_wait_intr+0xf0/0xf0 [11348.422781] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11348.434186] mdc_close+0x53b/0xd70 [mdc] [11348.443450] lmv_close+0x18d/0x490 [lmv] [11348.452393] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11348.459196] ll_md_real_close+0xa5/0x2a0 [lustre] [11348.470589] ll_md_close+0x274/0x800 [lustre] [11348.472287] ll_file_release+0x128/0xc50 [lustre] [11348.483284] __fput+0x10d/0x440 [11348.490121] ____fput+0x12/0x20 [11348.491935] task_work_run+0xc6/0x110 [11348.498135] exit_to_usermode_loop+0x1dd/0x1f0 [11348.503766] do_syscall_64+0x3d6/0x440 [11348.506445] entry_SYSCALL_64_after_hwframe+0x49/0xae [11348.509714] RIP: 0033:0x7f1a63983cd8 [11348.511738] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11348.534757] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11348.538889] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11348.542522] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11348.546193] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11348.550922] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11348.555154] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11352.056128] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11352.060881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11352.134441] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11352.147276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11352.170062] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11352.177194] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11352.199119] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11352.207364] RAX: 0000000000003087 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11352.220429] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11352.230616] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11352.235623] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11352.245616] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11352.253756] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11352.265803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11352.273351] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11352.286292] Call Trace: [11352.292480] [11352.293256] ? show_regs.cold.9+0x22/0x2f [11352.301616] ? watchdog_timer_fn.cold.6+0x62/0x10a [11352.316405] ? __hrtimer_run_queues+0x1c2/0x4d0 [11352.318852] ? watchdog+0x40/0x40 [11352.326205] ? hrtimer_interrupt+0x128/0x2c0 [11352.329684] ? smp_apic_timer_interrupt+0xa6/0x270 [11352.340801] ? apic_timer_interrupt+0xf/0x20 [11352.344155] [11352.351087] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11352.361154] do_raw_spin_lock+0x177/0x190 [11352.363311] _raw_spin_lock+0x21/0x30 [11352.371472] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11352.374854] ? do_raw_spin_unlock+0x75/0x190 [11352.381626] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11352.389026] ? do_wait_intr+0xf0/0xf0 [11352.393698] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11352.401169] mdc_close+0x53b/0xd70 [mdc] [11352.407735] lmv_close+0x18d/0x490 [lmv] [11352.419847] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11352.431487] ll_md_real_close+0xa5/0x2a0 [lustre] [11352.437834] ll_md_close+0x274/0x800 [lustre] [11352.446754] ll_file_release+0x128/0xc50 [lustre] [11352.450939] __fput+0x10d/0x440 [11352.460932] ____fput+0x12/0x20 [11352.463773] task_work_run+0xc6/0x110 [11352.472020] exit_to_usermode_loop+0x1dd/0x1f0 [11352.475530] do_syscall_64+0x3d6/0x440 [11352.481775] entry_SYSCALL_64_after_hwframe+0x49/0xae [11352.492075] RIP: 0033:0x7f1a63983cd8 [11352.499538] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11352.525073] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11352.547385] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11352.567086] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11352.585638] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11352.606463] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11352.627756] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11364.053767] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11364.063531] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11364.096050] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11364.103274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11364.110453] RIP: 0010:native_safe_halt+0x2e/0x50 [11364.112976] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11364.125087] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11364.129344] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11364.133266] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11364.141293] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11364.147159] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11364.150589] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11364.161029] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11364.172517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11364.184689] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11364.201615] Call Trace: [11364.210825] [11364.214335] ? show_regs.cold.9+0x22/0x2f [11364.223610] ? watchdog_timer_fn.cold.6+0x62/0x10a [11364.231610] ? __hrtimer_run_queues+0x1c2/0x4d0 [11364.235802] ? watchdog+0x40/0x40 [11364.238841] ? hrtimer_interrupt+0x128/0x2c0 [11364.242318] ? smp_apic_timer_interrupt+0xa6/0x270 [11364.244862] ? apic_timer_interrupt+0xf/0x20 [11364.246973] [11364.247908] ? native_safe_halt+0x2e/0x50 [11364.249769] kvm_wait+0xcd/0xe0 [11364.250832] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11364.252599] do_raw_spin_lock+0x177/0x190 [11364.254663] _raw_spin_lock+0x21/0x30 [11364.256208] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11364.264417] ? do_raw_spin_unlock+0x75/0x190 [11364.266894] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11364.271990] ? do_wait_intr+0xf0/0xf0 [11364.274182] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11364.277759] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11364.280696] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11364.283776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11364.285902] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11364.289036] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11364.291442] mdc_intent_lock+0x353/0x7e0 [mdc] [11364.293676] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11364.299866] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11364.306670] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11364.309170] lmv_intent_open+0x338/0x1150 [lmv] [11364.312825] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11364.318841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11364.322835] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11364.325923] ll_intent_lock+0xc5/0xaa0 [lustre] [11364.327570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11364.330198] ll_lookup_it+0xe0e/0x2270 [lustre] [11364.332947] ? do_raw_spin_unlock+0x75/0x190 [11364.335140] ? _raw_spin_unlock+0x12/0x30 [11364.337637] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11364.345565] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11364.352327] ll_atomic_open+0x36a/0xd70 [lustre] [11364.354718] lookup_open+0x356/0xb40 [11364.357327] do_last+0x313/0xfc0 [11364.358654] ? nd_jump_root+0xe5/0x160 [11364.360686] ? path_init+0x437/0x520 [11364.362216] path_openat+0xf7/0x500 [11364.364919] ? do_raw_spin_unlock+0x75/0x190 [11364.366663] do_filp_open+0x99/0x140 [11364.370202] ? getname_flags+0x6e/0x330 [11364.371815] ? __check_object_size+0xff/0x256 [11364.375878] ? do_raw_spin_unlock+0x75/0x190 [11364.379964] ? _raw_spin_unlock+0x12/0x30 [11364.384157] do_sys_openat2+0x2b4/0x410 [11364.389198] do_sys_open+0x73/0xa0 [11364.394843] __x64_sys_openat+0x24/0x30 [11364.398987] do_syscall_64+0xc1/0x440 [11364.406168] entry_SYSCALL_64_after_hwframe+0x49/0xae [11364.413124] RIP: 0033:0x7f1a63983332 [11364.416483] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11364.429814] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11364.435844] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11364.439550] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11364.443361] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11364.447406] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11364.451275] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11368.929089] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9345s! [11368.932963] Showing busy workqueues and worker pools: [11368.935949] workqueue events: flags=0x0 [11368.937937] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11368.941855] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11368.946027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11368.949211] pending: sec_gc_main [ptlrpc] [11368.951979] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11368.955362] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11368.960392] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11368.963591] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11368.968240] workqueue events_long: flags=0x0 [11368.970640] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11368.974589] pending: lu_objects_flush [obdclass] [11368.976969] workqueue events_unbound: flags=0x2 [11368.979092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11368.981984] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11368.987181] workqueue events_power_efficient: flags=0x80 [11368.989874] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11368.993198] pending: neigh_periodic_work, do_cache_clean [11368.996026] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11368.998578] pending: neigh_periodic_work, check_lifetime [11369.000353] workqueue mm_percpu_wq: flags=0x8 [11369.001878] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11369.004902] pending: vmstat_update [11369.007030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11369.010018] pending: vmstat_update [11369.011385] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11369.014120] pending: vmstat_update [11369.016102] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11369.019192] pending: vmstat_update [11369.021121] workqueue writeback: flags=0x4a [11369.023732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11369.027133] pending: wb_workfn, wb_workfn [11369.028308] workqueue xprtiod: flags=0xa [11369.030191] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11369.032158] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11369.035488] workqueue ipv6_addrconf: flags=0x40008 [11369.037367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11369.045507] pending: addrconf_verify_work [11369.048822] workqueue knbd0-recv: flags=0x1a [11369.050556] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11369.053543] in-flight: 647:recv_work [11369.055116] workqueue ptlrpc_pinger: flags=0x2 [11369.057241] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11369.059952] pending: ptlrpc_pinger_main [ptlrpc] [11369.062315] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11359s workers=2 idle: 652 [11372.061850] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [11372.066617] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11372.109795] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11372.132268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11372.159535] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11372.178657] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11372.208634] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11372.222888] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11372.233680] RDX: 00000000000029f0 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11372.240745] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11372.251369] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11372.260739] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11372.265550] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11372.275771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11372.285650] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11372.302061] Call Trace: [11372.304683] [11372.308291] ? show_regs.cold.9+0x22/0x2f [11372.318674] ? watchdog_timer_fn.cold.6+0x62/0x10a [11372.326467] ? __hrtimer_run_queues+0x1c2/0x4d0 [11372.337640] ? watchdog+0x40/0x40 [11372.341040] ? hrtimer_interrupt+0x128/0x2c0 [11372.353201] ? smp_apic_timer_interrupt+0xa6/0x270 [11372.359308] ? apic_timer_interrupt+0xf/0x20 [11372.366388] [11372.371998] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11372.382300] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11372.395542] do_raw_spin_lock+0x177/0x190 [11372.400506] _raw_spin_lock+0x21/0x30 [11372.403932] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11372.418035] obd_mod_free+0x46/0x140 [ptlrpc] [11372.432197] mdc_commit_open+0x91/0xb0 [mdc] [11372.441895] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11372.452735] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11372.463587] after_reply+0xe2c/0x1780 [ptlrpc] [11372.472980] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11372.486084] ? __next_timer_interrupt+0x160/0x160 [11372.495531] ptlrpcd+0xa82/0xe40 [ptlrpc] [11372.500783] ? do_wait_intr+0xf0/0xf0 [11372.502926] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11372.506697] kthread+0x1d1/0x200 [11372.508606] ? set_kthread_struct+0x70/0x70 [11372.510784] ret_from_fork+0x1f/0x30 [11376.057021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11376.061948] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11376.096196] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11376.107217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11376.112151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11376.115214] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11376.128894] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11376.133386] RAX: 0000000000006d88 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11376.137406] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11376.143384] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11376.147049] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11376.153668] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11376.157972] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11376.162233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11376.165788] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11376.169259] Call Trace: [11376.170787] [11376.171941] ? show_regs.cold.9+0x22/0x2f [11376.173882] ? watchdog_timer_fn.cold.6+0x62/0x10a [11376.186222] ? __hrtimer_run_queues+0x1c2/0x4d0 [11376.190140] ? watchdog+0x40/0x40 [11376.192892] ? hrtimer_interrupt+0x128/0x2c0 [11376.196746] ? smp_apic_timer_interrupt+0xa6/0x270 [11376.201061] ? apic_timer_interrupt+0xf/0x20 [11376.205105] [11376.206209] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11376.212749] do_raw_spin_lock+0x177/0x190 [11376.214786] _raw_spin_lock+0x21/0x30 [11376.217746] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11376.221024] ? do_raw_spin_unlock+0x75/0x190 [11376.223561] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11376.226193] ? do_wait_intr+0xf0/0xf0 [11376.227896] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11376.231149] mdc_close+0x53b/0xd70 [mdc] [11376.233334] lmv_close+0x18d/0x490 [lmv] [11376.235104] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11376.238038] ll_md_real_close+0xa5/0x2a0 [lustre] [11376.240497] ll_md_close+0x274/0x800 [lustre] [11376.242422] ll_file_release+0x128/0xc50 [lustre] [11376.244924] __fput+0x10d/0x440 [11376.246240] ____fput+0x12/0x20 [11376.247793] task_work_run+0xc6/0x110 [11376.249418] exit_to_usermode_loop+0x1dd/0x1f0 [11376.251730] do_syscall_64+0x3d6/0x440 [11376.253454] entry_SYSCALL_64_after_hwframe+0x49/0xae [11376.255518] RIP: 0033:0x7f1a63983cd8 [11376.257305] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11376.265346] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11376.269033] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11376.272412] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11376.275689] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11376.279190] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11376.282051] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11380.060134] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11380.069394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11380.127676] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11380.141280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11380.149858] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11380.159217] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11380.190469] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11380.201513] RAX: 0000000000007d88 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11380.217416] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11380.225367] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11380.232752] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11380.240213] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11380.250405] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11380.259840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11380.266128] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11380.273771] Call Trace: [11380.276734] [11380.277957] ? show_regs.cold.9+0x22/0x2f [11380.280759] ? watchdog_timer_fn.cold.6+0x62/0x10a [11380.284459] ? __hrtimer_run_queues+0x1c2/0x4d0 [11380.287111] ? watchdog+0x40/0x40 [11380.289062] ? hrtimer_interrupt+0x128/0x2c0 [11380.294017] ? smp_apic_timer_interrupt+0xa6/0x270 [11380.298754] ? apic_timer_interrupt+0xf/0x20 [11380.302989] [11380.304852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11380.313734] do_raw_spin_lock+0x177/0x190 [11380.317433] _raw_spin_lock+0x21/0x30 [11380.320364] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11380.332160] ? do_raw_spin_unlock+0x75/0x190 [11380.336317] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11380.347837] ? do_wait_intr+0xf0/0xf0 [11380.351698] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11380.370973] mdc_close+0x53b/0xd70 [mdc] [11380.377973] lmv_close+0x18d/0x490 [lmv] [11380.383220] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11380.398885] ll_md_real_close+0xa5/0x2a0 [lustre] [11380.406153] ll_md_close+0x274/0x800 [lustre] [11380.414062] ll_file_release+0x128/0xc50 [lustre] [11380.419346] __fput+0x10d/0x440 [11380.424233] ____fput+0x12/0x20 [11380.426260] task_work_run+0xc6/0x110 [11380.428284] exit_to_usermode_loop+0x1dd/0x1f0 [11380.434603] do_syscall_64+0x3d6/0x440 [11380.437590] entry_SYSCALL_64_after_hwframe+0x49/0xae [11380.440427] RIP: 0033:0x7f1a63983cd8 [11380.447526] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11380.458427] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11380.469130] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11380.475884] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11380.481075] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11380.491472] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11380.501108] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11396.058307] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11396.068277] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11396.148318] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11396.162372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11396.166983] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11396.171664] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11396.194315] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11396.210108] RAX: 000000000000410a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11396.221640] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11396.224298] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11396.238812] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11396.248610] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11396.261366] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11396.273033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11396.279571] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11396.286526] Call Trace: [11396.287992] [11396.288949] ? show_regs.cold.9+0x22/0x2f [11396.295207] ? watchdog_timer_fn.cold.6+0x62/0x10a [11396.296923] ? __hrtimer_run_queues+0x1c2/0x4d0 [11396.303644] ? watchdog+0x40/0x40 [11396.305194] ? hrtimer_interrupt+0x128/0x2c0 [11396.310978] ? smp_apic_timer_interrupt+0xa6/0x270 [11396.318337] ? apic_timer_interrupt+0xf/0x20 [11396.324881] [11396.330138] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11396.332155] do_raw_spin_lock+0x177/0x190 [11396.338461] _raw_spin_lock+0x21/0x30 [11396.340655] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11396.346234] ? do_raw_spin_unlock+0x75/0x190 [11396.361412] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11396.371530] ? do_wait_intr+0xf0/0xf0 [11396.376182] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11396.383817] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11396.391226] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11396.400329] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11396.406599] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11396.413312] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11396.415750] mdc_intent_lock+0x353/0x7e0 [mdc] [11396.421396] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11396.428129] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11396.430821] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11396.440330] lmv_intent_open+0x338/0x1150 [lmv] [11396.444930] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11396.450582] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11396.453531] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11396.458866] ll_intent_lock+0xc5/0xaa0 [lustre] [11396.460995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11396.467640] ll_lookup_it+0xe0e/0x2270 [lustre] [11396.472429] ? do_raw_spin_unlock+0x75/0x190 [11396.478134] ? _raw_spin_unlock+0x12/0x30 [11396.479897] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11396.482748] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11396.485172] ll_atomic_open+0x36a/0xd70 [lustre] [11396.487959] lookup_open+0x356/0xb40 [11396.493145] do_last+0x313/0xfc0 [11396.494692] ? nd_jump_root+0xe5/0x160 [11396.496883] ? path_init+0x437/0x520 [11396.500550] path_openat+0xf7/0x500 [11396.501900] ? do_raw_spin_unlock+0x75/0x190 [11396.507359] do_filp_open+0x99/0x140 [11396.509921] ? getname_flags+0x6e/0x330 [11396.513333] ? __check_object_size+0xff/0x256 [11396.518654] ? do_raw_spin_unlock+0x75/0x190 [11396.522890] ? _raw_spin_unlock+0x12/0x30 [11396.526052] do_sys_openat2+0x2b4/0x410 [11396.529686] do_sys_open+0x73/0xa0 [11396.533719] __x64_sys_openat+0x24/0x30 [11396.537346] do_syscall_64+0xc1/0x440 [11396.541093] entry_SYSCALL_64_after_hwframe+0x49/0xae [11396.544658] RIP: 0033:0x7f1a63983332 [11396.549841] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11396.565649] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11396.572991] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11396.579809] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11396.584846] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11396.588988] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11396.596493] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11399.137522] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9376s! [11399.150632] Showing busy workqueues and worker pools: [11399.157267] workqueue events: flags=0x0 [11399.162493] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11399.170241] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11399.183272] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11399.199173] pending: sec_gc_main [ptlrpc] [11399.210802] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11399.233574] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11399.243491] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11399.249441] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11399.257219] workqueue events_long: flags=0x0 [11399.262935] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11399.279738] pending: lu_objects_flush [obdclass] [11399.295525] workqueue events_unbound: flags=0x2 [11399.297288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11399.338952] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11399.369189] workqueue events_power_efficient: flags=0x80 [11399.387158] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11399.420077] pending: neigh_periodic_work, do_cache_clean [11399.440466] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11399.481885] pending: neigh_periodic_work, check_lifetime [11399.490864] workqueue mm_percpu_wq: flags=0x8 [11399.502403] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11399.520382] pending: vmstat_update [11399.526827] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11399.536873] pending: vmstat_update [11399.538277] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11399.542680] pending: vmstat_update [11399.545540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11399.554605] pending: vmstat_update [11399.559957] workqueue writeback: flags=0x4a [11399.564388] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11399.574983] pending: wb_workfn, wb_workfn [11399.587339] workqueue xprtiod: flags=0xa [11399.593596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11399.603764] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11399.608817] workqueue ipv6_addrconf: flags=0x40008 [11399.611247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11399.614016] pending: addrconf_verify_work [11399.615926] workqueue knbd0-recv: flags=0x1a [11399.617677] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11399.623056] in-flight: 647:recv_work [11399.628161] workqueue ptlrpc_pinger: flags=0x2 [11399.634821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11399.639844] pending: ptlrpc_pinger_main [ptlrpc] [11399.648493] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11389s workers=2 idle: 652 [11400.058027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [11400.065444] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11400.084917] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11400.089470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11400.093283] RIP: 0010:native_safe_halt+0x2e/0x50 [11400.095095] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11400.102605] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11400.105513] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11400.108686] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11400.111595] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11400.113925] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11400.116857] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11400.119505] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11400.122502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11400.124771] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11400.127832] Call Trace: [11400.128889] [11400.129656] ? show_regs.cold.9+0x22/0x2f [11400.131074] ? watchdog_timer_fn.cold.6+0x62/0x10a [11400.133451] ? __hrtimer_run_queues+0x1c2/0x4d0 [11400.135068] ? watchdog+0x40/0x40 [11400.136417] ? hrtimer_interrupt+0x128/0x2c0 [11400.138032] ? smp_apic_timer_interrupt+0xa6/0x270 [11400.140020] ? apic_timer_interrupt+0xf/0x20 [11400.141647] [11400.142711] ? native_safe_halt+0x2e/0x50 [11400.144336] kvm_wait+0xcd/0xe0 [11400.145298] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11400.147438] do_raw_spin_lock+0x177/0x190 [11400.149451] _raw_spin_lock+0x21/0x30 [11400.151035] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11400.153781] obd_mod_free+0x46/0x140 [ptlrpc] [11400.156653] mdc_commit_open+0x91/0xb0 [mdc] [11400.158610] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11400.161297] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11400.163818] after_reply+0xe2c/0x1780 [ptlrpc] [11400.166473] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11400.168765] ? __next_timer_interrupt+0x160/0x160 [11400.170460] ptlrpcd+0xa82/0xe40 [ptlrpc] [11400.172091] ? do_wait_intr+0xf0/0xf0 [11400.173800] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11400.176440] kthread+0x1d1/0x200 [11400.177872] ? set_kthread_struct+0x70/0x70 [11400.179849] ret_from_fork+0x1f/0x30 [11404.062770] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11404.076515] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11404.126375] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11404.131991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11404.138564] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11404.145848] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11404.162609] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11404.169549] RAX: 0000000000006807 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11404.176824] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11404.183753] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11404.189442] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11404.199600] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11404.205096] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11404.212554] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11404.218899] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11404.224788] Call Trace: [11404.225689] [11404.226423] ? show_regs.cold.9+0x22/0x2f [11404.229273] ? watchdog_timer_fn.cold.6+0x62/0x10a [11404.232049] ? __hrtimer_run_queues+0x1c2/0x4d0 [11404.236514] ? watchdog+0x40/0x40 [11404.238698] ? hrtimer_interrupt+0x128/0x2c0 [11404.243061] ? smp_apic_timer_interrupt+0xa6/0x270 [11404.247598] ? apic_timer_interrupt+0xf/0x20 [11404.251454] [11404.252187] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11404.255044] do_raw_spin_lock+0x177/0x190 [11404.262514] _raw_spin_lock+0x21/0x30 [11404.264413] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11404.275379] ? do_raw_spin_unlock+0x75/0x190 [11404.279030] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11404.288189] ? do_wait_intr+0xf0/0xf0 [11404.290988] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11404.293512] mdc_close+0x53b/0xd70 [mdc] [11404.299256] lmv_close+0x18d/0x490 [lmv] [11404.302967] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11404.306389] ll_md_real_close+0xa5/0x2a0 [lustre] [11404.315156] ll_md_close+0x274/0x800 [lustre] [11404.323049] ll_file_release+0x128/0xc50 [lustre] [11404.329288] __fput+0x10d/0x440 [11404.331261] ____fput+0x12/0x20 [11404.332711] task_work_run+0xc6/0x110 [11404.335211] exit_to_usermode_loop+0x1dd/0x1f0 [11404.339887] do_syscall_64+0x3d6/0x440 [11404.345147] entry_SYSCALL_64_after_hwframe+0x49/0xae [11404.352293] RIP: 0033:0x7f1a63983cd8 [11404.358256] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11404.394918] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11404.402764] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11404.416603] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11404.429836] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11404.438629] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11404.445669] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11408.057761] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11408.071076] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11408.146417] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11408.165379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11408.192817] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11408.200376] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11408.218367] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11408.224337] RAX: 0000000000007985 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11408.228037] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11408.231209] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11408.234950] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11408.238354] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11408.241811] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11408.245692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11408.247962] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11408.251141] Call Trace: [11408.252806] [11408.255081] ? show_regs.cold.9+0x22/0x2f [11408.258878] ? watchdog_timer_fn.cold.6+0x62/0x10a [11408.261457] ? __hrtimer_run_queues+0x1c2/0x4d0 [11408.263871] ? watchdog+0x40/0x40 [11408.265794] ? hrtimer_interrupt+0x128/0x2c0 [11408.267886] ? smp_apic_timer_interrupt+0xa6/0x270 [11408.271123] ? apic_timer_interrupt+0xf/0x20 [11408.274028] [11408.275124] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11408.281541] do_raw_spin_lock+0x177/0x190 [11408.286840] _raw_spin_lock+0x21/0x30 [11408.289935] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11408.297235] ? do_raw_spin_unlock+0x75/0x190 [11408.303733] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11408.308041] ? do_wait_intr+0xf0/0xf0 [11408.314047] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11408.320867] mdc_close+0x53b/0xd70 [mdc] [11408.323707] lmv_close+0x18d/0x490 [lmv] [11408.330032] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11408.339593] ll_md_real_close+0xa5/0x2a0 [lustre] [11408.352148] ll_md_close+0x274/0x800 [lustre] [11408.355871] ll_file_release+0x128/0xc50 [lustre] [11408.364400] __fput+0x10d/0x440 [11408.367885] ____fput+0x12/0x20 [11408.371268] task_work_run+0xc6/0x110 [11408.375656] exit_to_usermode_loop+0x1dd/0x1f0 [11408.378436] do_syscall_64+0x3d6/0x440 [11408.380608] entry_SYSCALL_64_after_hwframe+0x49/0xae [11408.385195] RIP: 0033:0x7f1a63983cd8 [11408.387450] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11408.399107] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11408.403869] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11408.411774] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11408.416305] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11408.420675] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11408.424031] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11424.053101] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11424.057407] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11424.091217] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11424.097800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11424.102629] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11424.108726] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11424.119697] RSP: 0018:ffffc38541c37500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11424.125219] RAX: 0000000000004889 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11424.130198] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11424.135573] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11424.139459] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11424.143693] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11424.148793] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11424.153870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11424.156867] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11424.163447] Call Trace: [11424.164612] [11424.165754] ? show_regs.cold.9+0x22/0x2f [11424.169915] ? watchdog_timer_fn.cold.6+0x62/0x10a [11424.175282] ? __hrtimer_run_queues+0x1c2/0x4d0 [11424.177946] ? watchdog+0x40/0x40 [11424.180000] ? hrtimer_interrupt+0x128/0x2c0 [11424.182554] ? smp_apic_timer_interrupt+0xa6/0x270 [11424.185222] ? apic_timer_interrupt+0xf/0x20 [11424.187887] [11424.189450] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11424.192546] do_raw_spin_lock+0x177/0x190 [11424.194902] _raw_spin_lock+0x21/0x30 [11424.196435] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11424.199875] ? do_raw_spin_unlock+0x75/0x190 [11424.205025] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11424.208398] ? do_wait_intr+0xf0/0xf0 [11424.210692] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11424.213904] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11424.216985] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11424.220423] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11424.223716] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11424.226625] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11424.229776] mdc_intent_lock+0x353/0x7e0 [mdc] [11424.232643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11424.235691] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11424.239363] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11424.242841] lmv_intent_open+0x338/0x1150 [lmv] [11424.245693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11424.248750] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11424.251770] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11424.254709] ll_intent_lock+0xc5/0xaa0 [lustre] [11424.257746] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11424.260882] ll_lookup_it+0xe0e/0x2270 [lustre] [11424.263754] ? do_raw_spin_unlock+0x75/0x190 [11424.266016] ? _raw_spin_unlock+0x12/0x30 [11424.268609] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11424.271578] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11424.274341] ll_atomic_open+0x36a/0xd70 [lustre] [11424.277408] lookup_open+0x356/0xb40 [11424.279249] do_last+0x313/0xfc0 [11424.281862] ? nd_jump_root+0xe5/0x160 [11424.283924] ? path_init+0x437/0x520 [11424.286036] path_openat+0xf7/0x500 [11424.287979] ? do_raw_spin_unlock+0x75/0x190 [11424.290751] do_filp_open+0x99/0x140 [11424.292824] ? getname_flags+0x6e/0x330 [11424.295245] ? __check_object_size+0xff/0x256 [11424.297896] ? do_raw_spin_unlock+0x75/0x190 [11424.300463] ? _raw_spin_unlock+0x12/0x30 [11424.302715] do_sys_openat2+0x2b4/0x410 [11424.304892] do_sys_open+0x73/0xa0 [11424.307090] __x64_sys_openat+0x24/0x30 [11424.309164] do_syscall_64+0xc1/0x440 [11424.311410] entry_SYSCALL_64_after_hwframe+0x49/0xae [11424.313397] RIP: 0033:0x7f1a63983332 [11424.315921] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11424.327180] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11424.332870] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11424.337151] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11424.341497] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11424.345974] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11424.350502] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11428.060406] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [11428.074333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11428.149793] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11428.170897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11428.184806] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11428.195893] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11428.225616] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11428.237353] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11428.247827] RDX: 0000000000007604 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11428.255612] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11428.260389] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11428.277536] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11428.293037] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11428.315438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11428.323655] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11428.338598] Call Trace: [11428.342708] [11428.348359] ? show_regs.cold.9+0x22/0x2f [11428.349764] ? watchdog_timer_fn.cold.6+0x62/0x10a [11428.363474] ? __hrtimer_run_queues+0x1c2/0x4d0 [11428.376735] ? watchdog+0x40/0x40 [11428.387457] ? hrtimer_interrupt+0x128/0x2c0 [11428.398847] ? smp_apic_timer_interrupt+0xa6/0x270 [11428.401754] ? apic_timer_interrupt+0xf/0x20 [11428.403450] [11428.404327] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11428.406573] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11428.408641] do_raw_spin_lock+0x177/0x190 [11428.410254] _raw_spin_lock+0x21/0x30 [11428.411624] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11428.413817] obd_mod_free+0x46/0x140 [ptlrpc] [11428.416003] mdc_commit_open+0x91/0xb0 [mdc] [11428.417641] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11428.420003] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11428.422569] after_reply+0xe2c/0x1780 [ptlrpc] [11428.424851] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11428.427454] ? __next_timer_interrupt+0x160/0x160 [11428.429517] ptlrpcd+0xa82/0xe40 [ptlrpc] [11428.431555] ? do_wait_intr+0xf0/0xf0 [11428.433419] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11428.435913] kthread+0x1d1/0x200 [11428.439267] ? set_kthread_struct+0x70/0x70 [11428.442946] ret_from_fork+0x1f/0x30 [11429.859195] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9406s! [11429.870744] Showing busy workqueues and worker pools: [11429.874903] workqueue events: flags=0x0 [11429.882401] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11429.890435] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11429.893787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11429.898872] pending: sec_gc_main [ptlrpc] [11429.907456] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11429.909921] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11429.914960] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11429.918320] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11429.924889] workqueue events_long: flags=0x0 [11429.926709] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11429.930495] pending: lu_objects_flush [obdclass] [11429.932846] workqueue events_unbound: flags=0x2 [11429.935027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11429.937554] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11429.943460] workqueue events_power_efficient: flags=0x80 [11429.945856] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11429.949658] pending: neigh_periodic_work, do_cache_clean [11429.952144] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11429.958984] pending: neigh_periodic_work, check_lifetime [11429.961726] workqueue mm_percpu_wq: flags=0x8 [11429.965977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11429.969210] pending: vmstat_update [11429.970857] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11429.974048] pending: vmstat_update [11429.976050] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11429.980441] pending: vmstat_update [11429.982075] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11429.985328] pending: vmstat_update [11429.987337] workqueue writeback: flags=0x4a [11429.989177] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11429.991846] pending: wb_workfn, wb_workfn [11429.993832] workqueue xprtiod: flags=0xa [11429.996753] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11429.999756] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11430.006503] workqueue ipv6_addrconf: flags=0x40008 [11430.007866] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11430.013764] pending: addrconf_verify_work [11430.016571] workqueue knbd0-recv: flags=0x1a [11430.019845] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11430.023733] in-flight: 647:recv_work [11430.035354] workqueue ptlrpc_pinger: flags=0x2 [11430.037138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11430.041974] pending: ptlrpc_pinger_main [ptlrpc] [11430.043925] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11420s workers=2 idle: 652 [11432.057070] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11432.061984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11432.104875] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11432.115418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11432.120751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11432.124915] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11432.144833] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11432.151251] RAX: 00000000000078cb RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11432.159837] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11432.167082] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11432.178756] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11432.189947] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11432.201836] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11432.207440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11432.218458] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11432.227686] Call Trace: [11432.228773] [11432.230106] ? show_regs.cold.9+0x22/0x2f [11432.232727] ? watchdog_timer_fn.cold.6+0x62/0x10a [11432.234949] ? __hrtimer_run_queues+0x1c2/0x4d0 [11432.242101] ? watchdog+0x40/0x40 [11432.246657] ? hrtimer_interrupt+0x128/0x2c0 [11432.252780] ? smp_apic_timer_interrupt+0xa6/0x270 [11432.257507] ? apic_timer_interrupt+0xf/0x20 [11432.271643] [11432.276452] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11432.283755] do_raw_spin_lock+0x177/0x190 [11432.291853] _raw_spin_lock+0x21/0x30 [11432.295742] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11432.300193] ? do_raw_spin_unlock+0x75/0x190 [11432.311750] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11432.315400] ? do_wait_intr+0xf0/0xf0 [11432.318712] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11432.327456] mdc_close+0x53b/0xd70 [mdc] [11432.332503] lmv_close+0x18d/0x490 [lmv] [11432.339734] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11432.348029] ll_md_real_close+0xa5/0x2a0 [lustre] [11432.351995] ll_md_close+0x274/0x800 [lustre] [11432.363434] ll_file_release+0x128/0xc50 [lustre] [11432.369883] __fput+0x10d/0x440 [11432.371095] ____fput+0x12/0x20 [11432.373616] task_work_run+0xc6/0x110 [11432.379057] exit_to_usermode_loop+0x1dd/0x1f0 [11432.380878] do_syscall_64+0x3d6/0x440 [11432.387894] entry_SYSCALL_64_after_hwframe+0x49/0xae [11432.394653] RIP: 0033:0x7f1a63983cd8 [11432.395743] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11432.411709] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11432.418503] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11432.424822] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11432.429374] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11432.440110] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11432.443421] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11436.062199] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11436.066666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11436.090846] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11436.096760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11436.101289] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11436.104315] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11436.115054] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11436.119591] RAX: 0000000000005b7b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11436.124288] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11436.128636] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11436.133412] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11436.138366] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11436.143927] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11436.148779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11436.152067] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11436.156920] Call Trace: [11436.158273] [11436.159495] ? show_regs.cold.9+0x22/0x2f [11436.162098] ? watchdog_timer_fn.cold.6+0x62/0x10a [11436.165232] ? __hrtimer_run_queues+0x1c2/0x4d0 [11436.167782] ? watchdog+0x40/0x40 [11436.169801] ? hrtimer_interrupt+0x128/0x2c0 [11436.173633] ? smp_apic_timer_interrupt+0xa6/0x270 [11436.176465] ? apic_timer_interrupt+0xf/0x20 [11436.178692] [11436.179852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11436.182595] do_raw_spin_lock+0x177/0x190 [11436.185632] _raw_spin_lock+0x21/0x30 [11436.187464] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11436.191027] ? do_raw_spin_unlock+0x75/0x190 [11436.193795] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11436.196772] ? do_wait_intr+0xf0/0xf0 [11436.198645] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11436.201838] mdc_close+0x53b/0xd70 [mdc] [11436.204237] lmv_close+0x18d/0x490 [lmv] [11436.206358] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11436.209538] ll_md_real_close+0xa5/0x2a0 [lustre] [11436.212026] ll_md_close+0x274/0x800 [lustre] [11436.216585] ll_file_release+0x128/0xc50 [lustre] [11436.218840] __fput+0x10d/0x440 [11436.220947] ____fput+0x12/0x20 [11436.222469] task_work_run+0xc6/0x110 [11436.224584] exit_to_usermode_loop+0x1dd/0x1f0 [11436.228597] do_syscall_64+0x3d6/0x440 [11436.230922] entry_SYSCALL_64_after_hwframe+0x49/0xae [11436.233722] RIP: 0033:0x7f1a63983cd8 [11436.236312] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11436.262653] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11436.267486] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11436.275893] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11436.285865] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11436.298235] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11436.309896] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11442.658499] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [11442.665181] rcu: 3-...!: (5918556 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [11442.676219] (detected by 1, t=9420252 jiffies, g=269633, q=3389) [11442.679556] Sending NMI from CPU 1 to CPUs 3: [11442.686650] NMI backtrace for cpu 3 [11442.686659] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11442.686662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11442.686663] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11442.686665] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11442.686668] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [11442.686671] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11442.686672] RDX: 00000000000039b5 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11442.686674] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11442.686675] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11442.686676] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11442.686678] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11442.686679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11442.686680] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11442.686681] Call Trace: [11442.686682] [11442.686683] ? show_regs.cold.9+0x22/0x2f [11442.686684] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11442.686685] ? nmi_cpu_backtrace_handler+0x15/0x40 [11442.686687] ? nmi_handle+0xbd/0x210 [11442.686688] ? default_do_nmi+0x4b/0x1b0 [11442.686689] ? do_nmi+0x280/0x380 [11442.686690] ? end_repeat_nmi+0x16/0x1f [11442.686691] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11442.686693] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11442.686695] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11442.686696] [11442.686697] do_raw_spin_lock+0x177/0x190 [11442.686698] _raw_spin_lock+0x21/0x30 [11442.686699] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11442.686700] obd_mod_free+0x46/0x140 [ptlrpc] [11442.686702] mdc_commit_open+0x91/0xb0 [mdc] [11442.686703] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11442.686705] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11442.686706] after_reply+0xe2c/0x1780 [ptlrpc] [11442.686707] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11442.686708] ? __next_timer_interrupt+0x160/0x160 [11442.686709] ptlrpcd+0xa82/0xe40 [ptlrpc] [11442.686711] ? do_wait_intr+0xf0/0xf0 [11442.686712] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11442.686713] kthread+0x1d1/0x200 [11442.686714] ? set_kthread_struct+0x70/0x70 [11442.686715] ret_from_fork+0x1f/0x30 [11442.688204] rcu: rcu_sched kthread starved for 9419898 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [11442.934339] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11442.945547] rcu: RCU grace-period kthread stack dump: [11442.950539] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11442.961220] Call Trace: [11442.962442] __schedule+0x351/0xcb0 [11442.966371] schedule+0xc0/0x180 [11442.970159] schedule_timeout+0xb4/0x190 [11442.974301] ? do_raw_spin_unlock+0x75/0x190 [11442.977776] ? __next_timer_interrupt+0x160/0x160 [11442.982474] ? prepare_to_swait_event+0xf3/0x140 [11442.987721] rcu_gp_kthread+0x8dd/0x15b0 [11442.991451] ? force_qs_rnp+0x2c0/0x2c0 [11442.994784] kthread+0x1d1/0x200 [11442.997725] ? set_kthread_struct+0x70/0x70 [11443.000115] ret_from_fork+0x1f/0x30 [11443.005272] rcu: Stack dump where RCU GP kthread last ran: [11443.012176] Sending NMI from CPU 1 to CPUs 2: [11443.021692] NMI backtrace for cpu 2 [11443.021701] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11443.021703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11443.021704] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11443.021706] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11443.021708] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 [11443.021711] RAX: 000000000000390c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11443.021712] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11443.021713] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11443.021715] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11443.021716] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11443.021717] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11443.021718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11443.021719] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11443.021720] Call Trace: [11443.021721] [11443.021722] ? show_regs.cold.9+0x22/0x2f [11443.021724] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11443.021724] ? nmi_cpu_backtrace_handler+0x15/0x40 [11443.021725] ? nmi_handle+0xbd/0x210 [11443.021726] ? default_do_nmi+0x4b/0x1b0 [11443.021727] ? do_nmi+0x280/0x380 [11443.021729] ? end_repeat_nmi+0x16/0x1f [11443.021730] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11443.021731] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11443.021732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11443.021733] [11443.021734] do_raw_spin_lock+0x177/0x190 [11443.021735] _raw_spin_lock+0x21/0x30 [11443.021735] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11443.021737] ? do_raw_spin_unlock+0x75/0x190 [11443.021738] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11443.021738] ? do_wait_intr+0xf0/0xf0 [11443.021740] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11443.021741] mdc_close+0x53b/0xd70 [mdc] [11443.021742] lmv_close+0x18d/0x490 [lmv] [11443.021743] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11443.021744] ll_md_real_close+0xa5/0x2a0 [lustre] [11443.021745] ll_md_close+0x274/0x800 [lustre] [11443.021746] ll_file_release+0x128/0xc50 [lustre] [11443.021747] __fput+0x10d/0x440 [11443.021748] ____fput+0x12/0x20 [11443.021749] task_work_run+0xc6/0x110 [11443.021750] exit_to_usermode_loop+0x1dd/0x1f0 [11443.021751] do_syscall_64+0x3d6/0x440 [11443.021752] entry_SYSCALL_64_after_hwframe+0x49/0xae [11443.021753] RIP: 0033:0x7f1a63983cd8 [11443.021754] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11443.021756] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11443.022441] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11443.022444] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11443.022446] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11443.022447] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11443.022448] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11456.053029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [11456.059188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11456.085529] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11456.092697] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11456.097047] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11456.099720] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11456.108555] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11456.111979] RAX: 0000000000000485 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11456.115562] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11456.118337] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11456.122453] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11456.124954] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11456.132660] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11456.136287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11456.137755] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11456.144632] Call Trace: [11456.147911] [11456.152701] ? show_regs.cold.9+0x22/0x2f [11456.154937] ? watchdog_timer_fn.cold.6+0x62/0x10a [11456.158768] ? __hrtimer_run_queues+0x1c2/0x4d0 [11456.160502] ? watchdog+0x40/0x40 [11456.161689] ? hrtimer_interrupt+0x128/0x2c0 [11456.163520] ? smp_apic_timer_interrupt+0xa6/0x270 [11456.165581] ? apic_timer_interrupt+0xf/0x20 [11456.167031] [11456.167694] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11456.169478] do_raw_spin_lock+0x177/0x190 [11456.170964] _raw_spin_lock+0x21/0x30 [11456.172075] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11456.173793] ? do_raw_spin_unlock+0x75/0x190 [11456.175421] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11456.177755] ? do_wait_intr+0xf0/0xf0 [11456.179464] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11456.181654] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11456.184072] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11456.186303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11456.188255] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11456.191468] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11456.193370] mdc_intent_lock+0x353/0x7e0 [mdc] [11456.197531] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11456.199632] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11456.210258] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11456.215758] lmv_intent_open+0x338/0x1150 [lmv] [11456.224347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11456.227204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11456.231186] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11456.235045] ll_intent_lock+0xc5/0xaa0 [lustre] [11456.239911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11456.242735] ll_lookup_it+0xe0e/0x2270 [lustre] [11456.248751] ? do_raw_spin_unlock+0x75/0x190 [11456.253561] ? _raw_spin_unlock+0x12/0x30 [11456.256760] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11456.259265] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11456.261532] ll_atomic_open+0x36a/0xd70 [lustre] [11456.264661] lookup_open+0x356/0xb40 [11456.266579] do_last+0x313/0xfc0 [11456.272703] ? nd_jump_root+0xe5/0x160 [11456.276537] ? path_init+0x437/0x520 [11456.278425] path_openat+0xf7/0x500 [11456.279793] ? do_raw_spin_unlock+0x75/0x190 [11456.282392] do_filp_open+0x99/0x140 [11456.284755] ? getname_flags+0x6e/0x330 [11456.290555] ? __check_object_size+0xff/0x256 [11456.296908] ? do_raw_spin_unlock+0x75/0x190 [11456.298449] ? _raw_spin_unlock+0x12/0x30 [11456.302534] do_sys_openat2+0x2b4/0x410 [11456.304598] do_sys_open+0x73/0xa0 [11456.307870] __x64_sys_openat+0x24/0x30 [11456.313572] do_syscall_64+0xc1/0x440 [11456.315593] entry_SYSCALL_64_after_hwframe+0x49/0xae [11456.320527] RIP: 0033:0x7f1a63983332 [11456.324391] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11456.336950] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11456.340323] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11456.343720] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11456.347473] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11456.351610] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11456.356195] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11460.071434] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9437s! [11460.082717] Showing busy workqueues and worker pools: [11460.086299] workqueue events: flags=0x0 [11460.090293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11460.098105] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11460.110462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11460.121341] pending: sec_gc_main [ptlrpc] [11460.130662] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11460.138473] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11460.152599] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11460.160536] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11460.171953] workqueue events_long: flags=0x0 [11460.176453] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11460.185414] pending: lu_objects_flush [obdclass] [11460.193640] workqueue events_unbound: flags=0x2 [11460.203561] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11460.214146] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11460.230301] workqueue events_power_efficient: flags=0x80 [11460.239429] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11460.243547] pending: neigh_periodic_work, do_cache_clean [11460.250343] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11460.256759] pending: neigh_periodic_work, check_lifetime [11460.259535] workqueue mm_percpu_wq: flags=0x8 [11460.264488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11460.271430] pending: vmstat_update [11460.275343] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11460.282923] pending: vmstat_update [11460.285644] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11460.293324] pending: vmstat_update [11460.296236] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11460.305339] pending: vmstat_update [11460.306737] workqueue writeback: flags=0x4a [11460.311888] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11460.322996] pending: wb_workfn, wb_workfn [11460.329728] workqueue xprtiod: flags=0xa [11460.335425] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11460.337706] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11460.351290] workqueue ipv6_addrconf: flags=0x40008 [11460.364350] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11460.382551] pending: addrconf_verify_work [11460.394881] workqueue knbd0-recv: flags=0x1a [11460.405250] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11460.415608] in-flight: 647:recv_work [11460.423526] workqueue ptlrpc_pinger: flags=0x2 [11460.429670] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11460.438918] pending: ptlrpc_pinger_main [ptlrpc] [11460.449888] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11450s workers=2 idle: 652 [11468.057055] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11468.057650] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11468.058139] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11468.058146] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11468.058224] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11468.058229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11468.058232] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11468.058244] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11468.058248] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11468.058252] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11468.058255] RDX: 000000000000641a RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11468.058256] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11468.058258] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11468.058260] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11468.058263] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11468.058266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11468.058268] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11468.058276] Call Trace: [11468.058282] [11468.058287] ? show_regs.cold.9+0x22/0x2f [11468.058297] ? watchdog_timer_fn.cold.6+0x62/0x10a [11468.058303] ? __hrtimer_run_queues+0x1c2/0x4d0 [11468.058307] ? watchdog+0x40/0x40 [11468.058312] ? hrtimer_interrupt+0x128/0x2c0 [11468.058316] ? smp_apic_timer_interrupt+0xa6/0x270 [11468.058324] ? apic_timer_interrupt+0xf/0x20 [11468.058329] [11468.058332] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11468.058337] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11468.058342] do_raw_spin_lock+0x177/0x190 [11468.058347] _raw_spin_lock+0x21/0x30 [11468.058352] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11468.058619] obd_mod_free+0x46/0x140 [ptlrpc] [11468.058829] mdc_commit_open+0x91/0xb0 [mdc] [11468.058862] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11468.059068] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11468.059261] after_reply+0xe2c/0x1780 [ptlrpc] [11468.059465] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11468.059665] ? __next_timer_interrupt+0x160/0x160 [11468.060362] ptlrpcd+0xa82/0xe40 [ptlrpc] [11468.060582] ? do_wait_intr+0xf0/0xf0 [11468.060591] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11468.060807] kthread+0x1d1/0x200 [11468.060812] ? set_kthread_struct+0x70/0x70 [11468.060817] ret_from_fork+0x1f/0x30 [11468.072965] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11468.073051] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11468.073059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11468.073062] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11468.073080] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11468.073085] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11468.073089] RAX: 0000000000005885 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11468.073092] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11468.073094] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11468.073096] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11468.073099] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11468.073102] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11468.073105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11468.073108] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11468.073117] Call Trace: [11468.073122] [11468.073128] ? show_regs.cold.9+0x22/0x2f [11468.073137] ? watchdog_timer_fn.cold.6+0x62/0x10a [11468.074193] ? __hrtimer_run_queues+0x1c2/0x4d0 [11468.074197] ? watchdog+0x40/0x40 [11468.074201] ? hrtimer_interrupt+0x128/0x2c0 [11468.074205] ? smp_apic_timer_interrupt+0xa6/0x270 [11468.074216] ? apic_timer_interrupt+0xf/0x20 [11468.074220] [11468.074223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11468.074230] do_raw_spin_lock+0x177/0x190 [11468.074236] _raw_spin_lock+0x21/0x30 [11468.074241] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11468.075071] ? do_raw_spin_unlock+0x75/0x190 [11468.075078] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11468.075276] ? do_wait_intr+0xf0/0xf0 [11468.075286] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11468.075490] mdc_close+0x53b/0xd70 [mdc] [11468.075529] lmv_close+0x18d/0x490 [lmv] [11468.075580] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11468.076483] ll_md_real_close+0xa5/0x2a0 [lustre] [11468.076576] ll_md_close+0x274/0x800 [lustre] [11468.076659] ll_file_release+0x128/0xc50 [lustre] [11468.085961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11468.086046] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11468.086054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11468.086058] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11468.086072] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11468.086077] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11468.086081] RAX: 0000000000006d83 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11468.086084] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11468.086087] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11468.086089] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11468.086091] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11468.086095] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11468.086098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11468.086100] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11468.086108] Call Trace: [11468.086113] [11468.086119] ? show_regs.cold.9+0x22/0x2f [11468.086140] ? watchdog_timer_fn.cold.6+0x62/0x10a [11468.086146] ? __hrtimer_run_queues+0x1c2/0x4d0 [11468.086150] ? watchdog+0x40/0x40 [11468.086155] ? hrtimer_interrupt+0x128/0x2c0 [11468.086159] ? smp_apic_timer_interrupt+0xa6/0x270 [11468.086167] ? apic_timer_interrupt+0xf/0x20 [11468.086171] [11468.086174] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11468.086180] do_raw_spin_lock+0x177/0x190 [11468.086184] _raw_spin_lock+0x21/0x30 [11468.086189] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11468.086465] ? do_raw_spin_unlock+0x75/0x190 [11468.086470] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11468.086673] ? do_wait_intr+0xf0/0xf0 [11468.086680] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11468.086903] mdc_close+0x53b/0xd70 [mdc] [11468.086939] lmv_close+0x18d/0x490 [lmv] [11468.086970] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11468.087090] ll_md_real_close+0xa5/0x2a0 [lustre] [11468.087218] ll_md_close+0x274/0x800 [lustre] [11468.087327] ll_file_release+0x128/0xc50 [lustre] [11468.098995] __fput+0x10d/0x440 [11468.099016] ____fput+0x12/0x20 [11468.099022] task_work_run+0xc6/0x110 [11468.099029] exit_to_usermode_loop+0x1dd/0x1f0 [11468.099036] do_syscall_64+0x3d6/0x440 [11468.099041] entry_SYSCALL_64_after_hwframe+0x49/0xae [11468.099048] RIP: 0033:0x7f1a63983cd8 [11468.099056] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11468.099060] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11468.099066] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11468.099068] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11468.099070] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11468.099072] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11468.099074] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11469.088328] __fput+0x10d/0x440 [11469.091500] ____fput+0x12/0x20 [11469.094030] task_work_run+0xc6/0x110 [11469.099586] exit_to_usermode_loop+0x1dd/0x1f0 [11469.102019] do_syscall_64+0x3d6/0x440 [11469.107026] entry_SYSCALL_64_after_hwframe+0x49/0xae [11469.113345] RIP: 0033:0x7f1a63983cd8 [11469.117073] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11469.135366] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11469.143993] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11469.151056] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11469.156376] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11469.162958] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11469.168982] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11488.053025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [11488.059918] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11488.087838] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11488.093911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11488.098471] RIP: 0010:native_safe_halt+0x2e/0x50 [11488.100817] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11488.114679] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11488.117919] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11488.120843] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [11488.124066] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11488.127454] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11488.130352] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11488.133372] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11488.137217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11488.139823] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11488.143055] Call Trace: [11488.144069] [11488.145055] ? show_regs.cold.9+0x22/0x2f [11488.147541] ? watchdog_timer_fn.cold.6+0x62/0x10a [11488.149411] ? __hrtimer_run_queues+0x1c2/0x4d0 [11488.151007] ? watchdog+0x40/0x40 [11488.152358] ? hrtimer_interrupt+0x128/0x2c0 [11488.154336] ? smp_apic_timer_interrupt+0xa6/0x270 [11488.156331] ? apic_timer_interrupt+0xf/0x20 [11488.158291] [11488.159327] ? native_safe_halt+0x2e/0x50 [11488.163050] kvm_wait+0xcd/0xe0 [11488.164351] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11488.167398] do_raw_spin_lock+0x177/0x190 [11488.169522] _raw_spin_lock+0x21/0x30 [11488.170723] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11488.173915] ? do_raw_spin_unlock+0x75/0x190 [11488.175952] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11488.180705] ? do_wait_intr+0xf0/0xf0 [11488.183908] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11488.186556] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11488.190649] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11488.193853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11488.196359] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11488.198029] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11488.202460] mdc_intent_lock+0x353/0x7e0 [mdc] [11488.204186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11488.207861] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11488.210969] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11488.213642] lmv_intent_open+0x338/0x1150 [lmv] [11488.216295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11488.218917] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11488.221933] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11488.223900] ll_intent_lock+0xc5/0xaa0 [lustre] [11488.226290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11488.228856] ll_lookup_it+0xe0e/0x2270 [lustre] [11488.231191] ? do_raw_spin_unlock+0x75/0x190 [11488.233083] ? _raw_spin_unlock+0x12/0x30 [11488.234939] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11488.236933] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11488.238256] ll_atomic_open+0x36a/0xd70 [lustre] [11488.239952] lookup_open+0x356/0xb40 [11488.241179] do_last+0x313/0xfc0 [11488.242646] ? nd_jump_root+0xe5/0x160 [11488.244012] ? path_init+0x437/0x520 [11488.245528] path_openat+0xf7/0x500 [11488.246694] ? do_raw_spin_unlock+0x75/0x190 [11488.248194] do_filp_open+0x99/0x140 [11488.249785] ? getname_flags+0x6e/0x330 [11488.251440] ? __check_object_size+0xff/0x256 [11488.252764] ? do_raw_spin_unlock+0x75/0x190 [11488.254345] ? _raw_spin_unlock+0x12/0x30 [11488.255977] do_sys_openat2+0x2b4/0x410 [11488.257604] do_sys_open+0x73/0xa0 [11488.258515] __x64_sys_openat+0x24/0x30 [11488.259595] do_syscall_64+0xc1/0x440 [11488.260946] entry_SYSCALL_64_after_hwframe+0x49/0xae [11488.262653] RIP: 0033:0x7f1a63983332 [11488.263681] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11488.269233] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11488.272376] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11488.275208] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11488.277410] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11488.279749] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11488.283884] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11490.784043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9467s! [11490.786555] Showing busy workqueues and worker pools: [11490.788881] workqueue events: flags=0x0 [11490.790686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11490.795847] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11490.799303] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11490.802987] pending: sec_gc_main [ptlrpc] [11490.805662] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11490.809095] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11490.813628] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11490.816708] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11490.820702] workqueue events_long: flags=0x0 [11490.822186] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11490.824526] pending: lu_objects_flush [obdclass] [11490.826401] workqueue events_unbound: flags=0x2 [11490.827961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11490.830432] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11490.834025] workqueue events_power_efficient: flags=0x80 [11490.835851] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11490.838662] pending: neigh_periodic_work, do_cache_clean [11490.840944] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11490.843711] pending: neigh_periodic_work, check_lifetime [11490.846206] workqueue mm_percpu_wq: flags=0x8 [11490.847837] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11490.850643] pending: vmstat_update [11490.852599] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11490.855549] pending: vmstat_update [11490.857294] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11490.859632] pending: vmstat_update [11490.861136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11490.864132] pending: vmstat_update [11490.865352] workqueue writeback: flags=0x4a [11490.867293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11490.869886] pending: wb_workfn, wb_workfn [11490.871779] workqueue xprtiod: flags=0xa [11490.873416] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11490.876023] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11490.879677] workqueue ipv6_addrconf: flags=0x40008 [11490.881703] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11490.884662] pending: addrconf_verify_work [11490.886676] workqueue knbd0-recv: flags=0x1a [11490.888669] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11490.891137] in-flight: 647:recv_work [11490.893080] workqueue ptlrpc_pinger: flags=0x2 [11490.895054] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11490.897818] pending: ptlrpc_pinger_main [ptlrpc] [11490.900604] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11480s workers=2 idle: 652 [11496.056086] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11496.059490] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11496.061130] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11496.063048] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11496.063124] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11496.063132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11496.063135] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11496.063151] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11496.063155] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11496.063159] RAX: 0000000000003be4 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11496.063161] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11496.063164] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11496.063166] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11496.063168] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11496.063171] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11496.063174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11496.063176] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11496.063186] Call Trace: [11496.063192] [11496.063198] ? show_regs.cold.9+0x22/0x2f [11496.063207] ? watchdog_timer_fn.cold.6+0x62/0x10a [11496.063213] ? __hrtimer_run_queues+0x1c2/0x4d0 [11496.063217] ? watchdog+0x40/0x40 [11496.063221] ? hrtimer_interrupt+0x128/0x2c0 [11496.063225] ? smp_apic_timer_interrupt+0xa6/0x270 [11496.063233] ? apic_timer_interrupt+0xf/0x20 [11496.063238] [11496.063241] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11496.063247] do_raw_spin_lock+0x177/0x190 [11496.063252] _raw_spin_lock+0x21/0x30 [11496.059417] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11496.063605] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11496.063610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11496.063612] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11496.063619] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11496.063623] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11496.063627] RAX: 0000000000002d08 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11496.063630] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11496.063632] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11496.063634] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11496.063636] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11496.063639] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11496.063642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11496.063649] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11496.063654] Call Trace: [11496.063658] [11496.063660] ? show_regs.cold.9+0x22/0x2f [11496.063666] ? watchdog_timer_fn.cold.6+0x62/0x10a [11496.063670] ? __hrtimer_run_queues+0x1c2/0x4d0 [11496.063673] ? watchdog+0x40/0x40 [11496.063677] ? hrtimer_interrupt+0x128/0x2c0 [11496.063681] ? smp_apic_timer_interrupt+0xa6/0x270 [11496.063686] ? apic_timer_interrupt+0xf/0x20 [11496.063690] [11496.063692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11496.063697] do_raw_spin_lock+0x177/0x190 [11496.063701] _raw_spin_lock+0x21/0x30 [11496.063704] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11496.064032] ? do_raw_spin_unlock+0x75/0x190 [11496.064038] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11496.064231] ? do_wait_intr+0xf0/0xf0 [11496.064239] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11496.064446] mdc_close+0x53b/0xd70 [mdc] [11496.064572] lmv_close+0x18d/0x490 [lmv] [11496.064603] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11496.064717] ll_md_real_close+0xa5/0x2a0 [lustre] [11496.064802] ll_md_close+0x274/0x800 [lustre] [11496.064885] ll_file_release+0x128/0xc50 [lustre] [11496.064971] __fput+0x10d/0x440 [11496.064980] ____fput+0x12/0x20 [11496.064984] task_work_run+0xc6/0x110 [11496.064991] exit_to_usermode_loop+0x1dd/0x1f0 [11496.064999] do_syscall_64+0x3d6/0x440 [11496.065004] entry_SYSCALL_64_after_hwframe+0x49/0xae [11496.065010] RIP: 0033:0x7f1a63983cd8 [11496.065017] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11496.065021] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11496.065025] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11496.065027] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11496.065029] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11496.065031] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11496.065034] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11496.063258] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11496.065543] ? do_raw_spin_unlock+0x75/0x190 [11496.065551] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11496.065761] ? do_wait_intr+0xf0/0xf0 [11496.065769] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11496.065974] mdc_close+0x53b/0xd70 [mdc] [11496.066019] lmv_close+0x18d/0x490 [lmv] [11496.066048] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11496.066230] ll_md_real_close+0xa5/0x2a0 [lustre] [11496.077852] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [11496.098600] ll_md_close+0x274/0x800 [lustre] [11496.098716] ll_file_release+0x128/0xc50 [lustre] [11496.098794] __fput+0x10d/0x440 [11496.098805] ____fput+0x12/0x20 [11496.098809] task_work_run+0xc6/0x110 [11496.098817] exit_to_usermode_loop+0x1dd/0x1f0 [11496.098824] do_syscall_64+0x3d6/0x440 [11496.098828] entry_SYSCALL_64_after_hwframe+0x49/0xae [11496.098836] RIP: 0033:0x7f1a63983cd8 [11496.098845] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11496.098849] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11496.098854] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11496.098857] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11496.098859] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11496.098862] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11496.098864] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11496.782702] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11496.813691] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11496.820251] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11496.828873] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11496.833642] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11496.856095] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11496.869902] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11496.884596] RDX: 0000000000006e6d RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11496.891931] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11496.898699] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11496.909018] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11496.920676] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11496.932804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11496.941982] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11496.954328] Call Trace: [11496.957910] [11496.960170] ? show_regs.cold.9+0x22/0x2f [11496.964948] ? watchdog_timer_fn.cold.6+0x62/0x10a [11496.974898] ? __hrtimer_run_queues+0x1c2/0x4d0 [11496.980107] ? watchdog+0x40/0x40 [11496.981651] ? hrtimer_interrupt+0x128/0x2c0 [11496.985195] ? smp_apic_timer_interrupt+0xa6/0x270 [11496.995150] ? apic_timer_interrupt+0xf/0x20 [11496.997137] [11496.999530] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11497.004183] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11497.008552] do_raw_spin_lock+0x177/0x190 [11497.010688] _raw_spin_lock+0x21/0x30 [11497.014007] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11497.018222] obd_mod_free+0x46/0x140 [ptlrpc] [11497.022699] mdc_commit_open+0x91/0xb0 [mdc] [11497.024921] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11497.028187] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11497.038982] after_reply+0xe2c/0x1780 [ptlrpc] [11497.043701] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11497.047003] ? __next_timer_interrupt+0x160/0x160 [11497.051351] ptlrpcd+0xa82/0xe40 [ptlrpc] [11497.056591] ? do_wait_intr+0xf0/0xf0 [11497.057965] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11497.062746] kthread+0x1d1/0x200 [11497.072233] ? set_kthread_struct+0x70/0x70 [11497.075242] ret_from_fork+0x1f/0x30 [11516.053333] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [11516.058636] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11516.100775] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11516.107377] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11516.114380] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11516.121478] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11516.140286] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11516.144459] RAX: 0000000000000583 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11516.149384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11516.156813] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11516.168072] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11516.174962] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11516.178919] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11516.185612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11516.189191] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11516.193251] Call Trace: [11516.194664] [11516.197422] ? show_regs.cold.9+0x22/0x2f [11516.199858] ? watchdog_timer_fn.cold.6+0x62/0x10a [11516.201687] ? __hrtimer_run_queues+0x1c2/0x4d0 [11516.207811] ? watchdog+0x40/0x40 [11516.214160] ? hrtimer_interrupt+0x128/0x2c0 [11516.224608] ? smp_apic_timer_interrupt+0xa6/0x270 [11516.227146] ? apic_timer_interrupt+0xf/0x20 [11516.231768] [11516.233139] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11516.237982] do_raw_spin_lock+0x177/0x190 [11516.240810] _raw_spin_lock+0x21/0x30 [11516.245147] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11516.249661] ? do_raw_spin_unlock+0x75/0x190 [11516.251716] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11516.256792] ? do_wait_intr+0xf0/0xf0 [11516.258585] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11516.263983] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11516.271155] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11516.275489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11516.278489] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11516.282184] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11516.284919] mdc_intent_lock+0x353/0x7e0 [mdc] [11516.289536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11516.292058] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11516.294756] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11516.298516] lmv_intent_open+0x338/0x1150 [lmv] [11516.311623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11516.319144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11516.333097] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11516.340666] ll_intent_lock+0xc5/0xaa0 [lustre] [11516.351122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11516.359040] ll_lookup_it+0xe0e/0x2270 [lustre] [11516.362420] ? do_raw_spin_unlock+0x75/0x190 [11516.365970] ? _raw_spin_unlock+0x12/0x30 [11516.368627] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11516.372383] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11516.383914] ll_atomic_open+0x36a/0xd70 [lustre] [11516.387714] lookup_open+0x356/0xb40 [11516.391445] do_last+0x313/0xfc0 [11516.394176] ? nd_jump_root+0xe5/0x160 [11516.397840] ? path_init+0x437/0x520 [11516.406592] path_openat+0xf7/0x500 [11516.414019] ? do_raw_spin_unlock+0x75/0x190 [11516.424031] do_filp_open+0x99/0x140 [11516.427727] ? getname_flags+0x6e/0x330 [11516.434192] ? __check_object_size+0xff/0x256 [11516.438186] ? do_raw_spin_unlock+0x75/0x190 [11516.442233] ? _raw_spin_unlock+0x12/0x30 [11516.444593] do_sys_openat2+0x2b4/0x410 [11516.450074] do_sys_open+0x73/0xa0 [11516.453769] __x64_sys_openat+0x24/0x30 [11516.456582] do_syscall_64+0xc1/0x440 [11516.460580] entry_SYSCALL_64_after_hwframe+0x49/0xae [11516.466032] RIP: 0033:0x7f1a63983332 [11516.470431] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11516.483507] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11516.487525] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11516.498485] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11516.504465] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11516.508263] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11516.511738] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11520.998480] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9497s! [11521.014046] Showing busy workqueues and worker pools: [11521.026552] workqueue events: flags=0x0 [11521.032584] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11521.040340] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11521.046943] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11521.053491] pending: sec_gc_main [ptlrpc] [11521.063488] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11521.072634] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11521.086088] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11521.096358] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11521.109236] workqueue events_long: flags=0x0 [11521.115436] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11521.121710] pending: lu_objects_flush [obdclass] [11521.129877] workqueue events_unbound: flags=0x2 [11521.136024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11521.144858] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11521.152667] workqueue events_power_efficient: flags=0x80 [11521.156463] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11521.166631] pending: neigh_periodic_work, do_cache_clean [11521.175796] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11521.184472] pending: neigh_periodic_work, check_lifetime [11521.196862] workqueue mm_percpu_wq: flags=0x8 [11521.209621] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11521.220296] pending: vmstat_update [11521.229510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11521.238150] pending: vmstat_update [11521.242083] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11521.251796] pending: vmstat_update [11521.256647] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11521.264591] pending: vmstat_update [11521.268804] workqueue writeback: flags=0x4a [11521.276769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11521.284682] pending: wb_workfn, wb_workfn [11521.291359] workqueue xprtiod: flags=0xa [11521.296605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11521.307012] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11521.318709] workqueue ipv6_addrconf: flags=0x40008 [11521.327972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11521.338996] pending: addrconf_verify_work [11521.348027] workqueue knbd0-recv: flags=0x1a [11521.353509] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11521.361013] in-flight: 647:recv_work [11521.365600] workqueue ptlrpc_pinger: flags=0x2 [11521.372808] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11521.385717] pending: ptlrpc_pinger_main [ptlrpc] [11521.395406] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11511s workers=2 idle: 652 [11524.057461] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11524.057780] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11524.058090] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11524.058097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11524.058180] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11524.058185] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11524.058188] RIP: 0010:native_safe_halt+0x2e/0x50 [11524.058201] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11524.058204] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11524.058209] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11524.058211] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11524.058213] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11524.058216] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11524.058219] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11524.058222] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11524.058225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11524.058227] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11524.058235] Call Trace: [11524.058241] [11524.058247] ? show_regs.cold.9+0x22/0x2f [11524.058257] ? watchdog_timer_fn.cold.6+0x62/0x10a [11524.058265] ? __hrtimer_run_queues+0x1c2/0x4d0 [11524.058269] ? watchdog+0x40/0x40 [11524.058273] ? hrtimer_interrupt+0x128/0x2c0 [11524.058278] ? smp_apic_timer_interrupt+0xa6/0x270 [11524.058284] ? apic_timer_interrupt+0xf/0x20 [11524.058290] [11524.058293] ? native_safe_halt+0x2e/0x50 [11524.058298] kvm_wait+0xcd/0xe0 [11524.058307] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11524.058315] do_raw_spin_lock+0x177/0x190 [11524.058319] _raw_spin_lock+0x21/0x30 [11524.058324] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11524.058752] obd_mod_free+0x46/0x140 [ptlrpc] [11524.058957] mdc_commit_open+0x91/0xb0 [mdc] [11524.058993] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11524.059414] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11524.059620] after_reply+0xe2c/0x1780 [ptlrpc] [11524.059897] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11524.060180] ? __next_timer_interrupt+0x160/0x160 [11524.060189] ptlrpcd+0xa82/0xe40 [ptlrpc] [11524.060478] ? do_wait_intr+0xf0/0xf0 [11524.060487] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11524.060703] kthread+0x1d1/0x200 [11524.060708] ? set_kthread_struct+0x70/0x70 [11524.060712] ret_from_fork+0x1f/0x30 [11524.073138] Modules linked in: lustre(O) [11524.080072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11524.080332] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11524.080340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11524.080344] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11524.080358] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11524.080363] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11524.080368] RAX: 0000000000006f89 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11524.080370] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11524.080373] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11524.080375] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11524.080377] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11524.080380] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11524.080383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11524.080385] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11524.080797] Call Trace: [11524.080804] [11524.080809] ? show_regs.cold.9+0x22/0x2f [11524.080820] ? watchdog_timer_fn.cold.6+0x62/0x10a [11524.080827] ? __hrtimer_run_queues+0x1c2/0x4d0 [11524.080831] ? watchdog+0x40/0x40 [11524.080835] ? hrtimer_interrupt+0x128/0x2c0 [11524.080839] ? smp_apic_timer_interrupt+0xa6/0x270 [11524.080847] ? apic_timer_interrupt+0xf/0x20 [11524.080852] [11524.080855] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11524.080862] do_raw_spin_lock+0x177/0x190 [11524.080867] _raw_spin_lock+0x21/0x30 [11524.080872] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11524.081955] ? do_raw_spin_unlock+0x75/0x190 [11524.081964] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11524.082928] ? do_wait_intr+0xf0/0xf0 [11524.082938] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11524.083830] mdc_close+0x53b/0xd70 [mdc] [11524.083871] lmv_close+0x18d/0x490 [lmv] [11524.083905] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11524.084662] ll_md_real_close+0xa5/0x2a0 [lustre] [11524.084764] ll_md_close+0x274/0x800 [lustre] [11524.084865] ll_file_release+0x128/0xc50 [lustre] [11524.084964] __fput+0x10d/0x440 [11524.085703] ____fput+0x12/0x20 [11524.085714] task_work_run+0xc6/0x110 [11524.085722] exit_to_usermode_loop+0x1dd/0x1f0 [11524.085776] do_syscall_64+0x3d6/0x440 [11524.085782] entry_SYSCALL_64_after_hwframe+0x49/0xae [11524.085791] RIP: 0033:0x7f1a63983cd8 [11524.085799] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11524.085804] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11524.085810] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11524.085812] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11524.085814] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11524.085816] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11524.085819] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11524.617981] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11524.652624] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11524.663082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11524.665966] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11524.669557] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11524.677308] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11524.680894] RAX: 0000000000002188 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11524.684570] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11524.687408] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11524.690286] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11524.693487] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11524.696654] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11524.704041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11524.708773] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11524.712633] Call Trace: [11524.713798] [11524.715043] ? show_regs.cold.9+0x22/0x2f [11524.717269] ? watchdog_timer_fn.cold.6+0x62/0x10a [11524.721708] ? __hrtimer_run_queues+0x1c2/0x4d0 [11524.726380] ? watchdog+0x40/0x40 [11524.730367] ? hrtimer_interrupt+0x128/0x2c0 [11524.737300] ? smp_apic_timer_interrupt+0xa6/0x270 [11524.741601] ? apic_timer_interrupt+0xf/0x20 [11524.747117] [11524.748045] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11524.751342] do_raw_spin_lock+0x177/0x190 [11524.756069] _raw_spin_lock+0x21/0x30 [11524.758115] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11524.769159] ? do_raw_spin_unlock+0x75/0x190 [11524.773722] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11524.787833] ? do_wait_intr+0xf0/0xf0 [11524.792912] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11524.801123] mdc_close+0x53b/0xd70 [mdc] [11524.806864] lmv_close+0x18d/0x490 [lmv] [11524.810359] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11524.814691] ll_md_real_close+0xa5/0x2a0 [lustre] [11524.821045] ll_md_close+0x274/0x800 [lustre] [11524.824907] ll_file_release+0x128/0xc50 [lustre] [11524.829912] __fput+0x10d/0x440 [11524.833531] ____fput+0x12/0x20 [11524.835968] task_work_run+0xc6/0x110 [11524.838159] exit_to_usermode_loop+0x1dd/0x1f0 [11524.840851] do_syscall_64+0x3d6/0x440 [11524.844212] entry_SYSCALL_64_after_hwframe+0x49/0xae [11524.848248] RIP: 0033:0x7f1a63983cd8 [11524.850576] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11524.887644] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11524.894333] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11524.897027] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11524.903104] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11524.907114] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11524.911405] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11548.053035] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11548.061416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11548.087643] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11548.095039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11548.099359] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11548.104061] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11548.114226] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11548.117888] RAX: 0000000000004e28 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11548.123502] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11548.128140] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11548.133521] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11548.137592] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11548.141222] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11548.144973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11548.147426] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11548.151510] Call Trace: [11548.153122] [11548.154353] ? show_regs.cold.9+0x22/0x2f [11548.157308] ? watchdog_timer_fn.cold.6+0x62/0x10a [11548.160565] ? __hrtimer_run_queues+0x1c2/0x4d0 [11548.164344] ? watchdog+0x40/0x40 [11548.165803] ? hrtimer_interrupt+0x128/0x2c0 [11548.169845] ? smp_apic_timer_interrupt+0xa6/0x270 [11548.172127] ? apic_timer_interrupt+0xf/0x20 [11548.174161] [11548.175139] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11548.177454] do_raw_spin_lock+0x177/0x190 [11548.181284] _raw_spin_lock+0x21/0x30 [11548.182837] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11548.185888] ? do_raw_spin_unlock+0x75/0x190 [11548.188365] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11548.190454] ? do_wait_intr+0xf0/0xf0 [11548.191855] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11548.194116] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11548.196268] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11548.199351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11548.204124] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11548.207215] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11548.210160] mdc_intent_lock+0x353/0x7e0 [mdc] [11548.212672] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11548.216344] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11548.222134] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11548.227509] lmv_intent_open+0x338/0x1150 [lmv] [11548.230664] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11548.234126] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11548.238886] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11548.241888] ll_intent_lock+0xc5/0xaa0 [lustre] [11548.244676] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11548.248792] ll_lookup_it+0xe0e/0x2270 [lustre] [11548.252081] ? do_raw_spin_unlock+0x75/0x190 [11548.254307] ? _raw_spin_unlock+0x12/0x30 [11548.257212] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11548.260524] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11548.263286] ll_atomic_open+0x36a/0xd70 [lustre] [11548.265856] lookup_open+0x356/0xb40 [11548.268333] do_last+0x313/0xfc0 [11548.270139] ? nd_jump_root+0xe5/0x160 [11548.271864] ? path_init+0x437/0x520 [11548.276165] path_openat+0xf7/0x500 [11548.278411] ? do_raw_spin_unlock+0x75/0x190 [11548.280578] do_filp_open+0x99/0x140 [11548.284377] ? getname_flags+0x6e/0x330 [11548.286603] ? __check_object_size+0xff/0x256 [11548.288014] ? do_raw_spin_unlock+0x75/0x190 [11548.289667] ? _raw_spin_unlock+0x12/0x30 [11548.291498] do_sys_openat2+0x2b4/0x410 [11548.293367] do_sys_open+0x73/0xa0 [11548.294896] __x64_sys_openat+0x24/0x30 [11548.296488] do_syscall_64+0xc1/0x440 [11548.298168] entry_SYSCALL_64_after_hwframe+0x49/0xae [11548.300272] RIP: 0033:0x7f1a63983332 [11548.301967] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11548.311219] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11548.317167] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11548.320365] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11548.324569] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11548.328688] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11548.332306] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11551.712400] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9528s! [11551.721430] Showing busy workqueues and worker pools: [11551.725689] workqueue events: flags=0x0 [11551.728271] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11551.738010] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11551.744026] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11551.748293] pending: sec_gc_main [ptlrpc] [11551.754613] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11551.759489] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11551.767095] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11551.771322] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11551.776191] workqueue events_long: flags=0x0 [11551.777986] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11551.783961] pending: lu_objects_flush [obdclass] [11551.787585] workqueue events_unbound: flags=0x2 [11551.791954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11551.795378] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11551.804461] workqueue events_power_efficient: flags=0x80 [11551.807425] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11551.813157] pending: neigh_periodic_work, do_cache_clean [11551.815489] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11551.821166] pending: neigh_periodic_work, check_lifetime [11551.823148] workqueue mm_percpu_wq: flags=0x8 [11551.824551] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11551.829255] pending: vmstat_update [11551.831225] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11551.834110] pending: vmstat_update [11551.835491] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11551.838307] pending: vmstat_update [11551.839947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11551.842716] pending: vmstat_update [11551.844074] workqueue writeback: flags=0x4a [11551.845567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11551.848296] pending: wb_workfn, wb_workfn [11551.850189] workqueue xprtiod: flags=0xa [11551.851952] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11551.854549] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11551.858012] workqueue ipv6_addrconf: flags=0x40008 [11551.860022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11551.862741] pending: addrconf_verify_work [11551.864662] workqueue knbd0-recv: flags=0x1a [11551.866575] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11551.869398] in-flight: 647:recv_work [11551.871170] workqueue ptlrpc_pinger: flags=0x2 [11551.872977] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11551.875486] pending: ptlrpc_pinger_main [ptlrpc] [11551.877988] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11541s workers=2 idle: 652 [11552.056049] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11552.057675] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11552.061542] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11552.061556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11552.061626] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11552.061631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11552.061634] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11552.061649] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11552.061654] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11552.061658] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11552.061660] RDX: 0000000000000351 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11552.061662] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11552.061664] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11552.061666] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11552.061669] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11552.061672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11552.061674] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11552.061683] Call Trace: [11552.061690] [11552.061707] ? show_regs.cold.9+0x22/0x2f [11552.061718] ? watchdog_timer_fn.cold.6+0x62/0x10a [11552.061724] ? __hrtimer_run_queues+0x1c2/0x4d0 [11552.061728] ? watchdog+0x40/0x40 [11552.061732] ? hrtimer_interrupt+0x128/0x2c0 [11552.061736] ? smp_apic_timer_interrupt+0xa6/0x270 [11552.061744] ? apic_timer_interrupt+0xf/0x20 [11552.061864] Modules linked in: lustre(O) [11552.061892] [11552.061895] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11552.061900] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11552.061904] do_raw_spin_lock+0x177/0x190 [11552.061908] _raw_spin_lock+0x21/0x30 [11552.061922] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11552.062624] obd_mod_free+0x46/0x140 [ptlrpc] [11552.062916] mdc_commit_open+0x91/0xb0 [mdc] [11552.062963] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11552.063385] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11552.063658] after_reply+0xe2c/0x1780 [ptlrpc] [11552.064023] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11552.064417] ? __next_timer_interrupt+0x160/0x160 [11552.064427] ptlrpcd+0xa82/0xe40 [ptlrpc] [11552.064705] ? do_wait_intr+0xf0/0xf0 [11552.064715] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11552.065270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [11552.065295] kthread+0x1d1/0x200 [11552.065299] lmv(O) ec(O) [11552.065306] ? set_kthread_struct+0x70/0x70 [11552.065309] fid(O) fld(O) ptlrpc_gss(O) [11552.065314] ret_from_fork+0x1f/0x30 [11552.065318] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11552.065371] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11552.065379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11552.070235] osp(O) [11552.102161] RIP: 0010:native_safe_halt+0x2e/0x50 [11552.102183] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11552.102187] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11552.102193] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11552.102195] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11552.102197] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11552.102199] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11552.102202] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11552.102205] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11552.102208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11552.102210] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11552.102218] Call Trace: [11552.102224] [11552.102230] ? show_regs.cold.9+0x22/0x2f [11552.102243] ? watchdog_timer_fn.cold.6+0x62/0x10a [11552.102249] ? __hrtimer_run_queues+0x1c2/0x4d0 [11552.102254] ? watchdog+0x40/0x40 [11552.102258] ? hrtimer_interrupt+0x128/0x2c0 [11552.102262] ? smp_apic_timer_interrupt+0xa6/0x270 [11552.102267] ? apic_timer_interrupt+0xf/0x20 [11552.102272] [11552.102276] ? native_safe_halt+0x2e/0x50 [11552.111495] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11552.111580] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11552.111587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11552.111589] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11552.111601] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11552.111604] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11552.111609] RAX: 0000000000002b88 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11552.111611] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11552.111613] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11552.111615] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11552.111617] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11552.111620] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11552.111623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11552.111625] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11552.111632] Call Trace: [11552.111636] [11552.111641] ? show_regs.cold.9+0x22/0x2f [11552.111650] ? watchdog_timer_fn.cold.6+0x62/0x10a [11552.111655] ? __hrtimer_run_queues+0x1c2/0x4d0 [11552.111658] ? watchdog+0x40/0x40 [11552.111662] ? hrtimer_interrupt+0x128/0x2c0 [11552.111665] ? smp_apic_timer_interrupt+0xa6/0x270 [11552.111672] ? apic_timer_interrupt+0xf/0x20 [11552.111676] [11552.111678] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11552.111683] do_raw_spin_lock+0x177/0x190 [11552.111687] _raw_spin_lock+0x21/0x30 [11552.111691] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11552.120010] kvm_wait+0xcd/0xe0 [11552.120033] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11552.120042] do_raw_spin_lock+0x177/0x190 [11552.120050] _raw_spin_lock+0x21/0x30 [11552.120057] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11552.120329] ? do_raw_spin_unlock+0x75/0x190 [11552.120335] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11552.120534] ? do_wait_intr+0xf0/0xf0 [11552.120545] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11552.120736] mdc_close+0x53b/0xd70 [mdc] [11552.120892] lmv_close+0x18d/0x490 [lmv] [11552.120927] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11552.121136] ll_md_real_close+0xa5/0x2a0 [lustre] [11552.121214] ll_md_close+0x274/0x800 [lustre] [11552.121290] ll_file_release+0x128/0xc50 [lustre] [11552.121365] __fput+0x10d/0x440 [11552.121373] ____fput+0x12/0x20 [11552.121376] task_work_run+0xc6/0x110 [11552.121383] exit_to_usermode_loop+0x1dd/0x1f0 [11552.121388] do_syscall_64+0x3d6/0x440 [11552.121392] entry_SYSCALL_64_after_hwframe+0x49/0xae [11552.121399] RIP: 0033:0x7f1a63983cd8 [11552.121406] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11552.121409] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11552.121414] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11552.121416] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11552.121418] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11552.121420] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11552.121422] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11552.435126] ? do_raw_spin_unlock+0x75/0x190 [11552.437003] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11552.439426] ? do_wait_intr+0xf0/0xf0 [11552.441053] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11552.443386] mdc_close+0x53b/0xd70 [mdc] [11552.445147] lmv_close+0x18d/0x490 [lmv] [11552.447291] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11552.452527] ll_md_real_close+0xa5/0x2a0 [lustre] [11552.454641] ll_md_close+0x274/0x800 [lustre] [11552.457162] ll_file_release+0x128/0xc50 [lustre] [11552.459778] __fput+0x10d/0x440 [11552.461791] ____fput+0x12/0x20 [11552.463287] task_work_run+0xc6/0x110 [11552.465524] exit_to_usermode_loop+0x1dd/0x1f0 [11552.467602] do_syscall_64+0x3d6/0x440 [11552.470366] entry_SYSCALL_64_after_hwframe+0x49/0xae [11552.473251] RIP: 0033:0x7f1a63983cd8 [11552.474457] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11552.481012] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11552.485456] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11552.489719] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11552.494336] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11552.497385] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11552.502285] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11576.053597] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [11576.056928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11576.075530] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11576.079704] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11576.083180] RIP: 0010:native_safe_halt+0x2e/0x50 [11576.085170] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11576.093537] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11576.096429] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11576.099022] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [11576.102056] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11576.104394] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11576.107246] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11576.112873] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11576.116516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11576.119047] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11576.122245] Call Trace: [11576.123402] [11576.124598] ? show_regs.cold.9+0x22/0x2f [11576.126219] ? watchdog_timer_fn.cold.6+0x62/0x10a [11576.127634] ? __hrtimer_run_queues+0x1c2/0x4d0 [11576.129262] ? watchdog+0x40/0x40 [11576.130769] ? hrtimer_interrupt+0x128/0x2c0 [11576.132235] ? smp_apic_timer_interrupt+0xa6/0x270 [11576.134336] ? apic_timer_interrupt+0xf/0x20 [11576.136187] [11576.137234] ? native_safe_halt+0x2e/0x50 [11576.139187] kvm_wait+0xcd/0xe0 [11576.140930] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11576.142937] do_raw_spin_lock+0x177/0x190 [11576.144350] _raw_spin_lock+0x21/0x30 [11576.145620] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11576.147623] ? do_raw_spin_unlock+0x75/0x190 [11576.149216] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11576.151198] ? do_wait_intr+0xf0/0xf0 [11576.152428] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11576.154503] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11576.156800] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11576.159099] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11576.161591] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11576.163854] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11576.166002] mdc_intent_lock+0x353/0x7e0 [mdc] [11576.167722] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11576.169487] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11576.171410] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11576.173381] lmv_intent_open+0x338/0x1150 [lmv] [11576.174932] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11576.176832] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11576.178743] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11576.180196] ll_intent_lock+0xc5/0xaa0 [lustre] [11576.181824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11576.183615] ll_lookup_it+0xe0e/0x2270 [lustre] [11576.185098] ? do_raw_spin_unlock+0x75/0x190 [11576.186670] ? _raw_spin_unlock+0x12/0x30 [11576.188147] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11576.190178] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11576.192341] ll_atomic_open+0x36a/0xd70 [lustre] [11576.194412] lookup_open+0x356/0xb40 [11576.195390] do_last+0x313/0xfc0 [11576.196433] ? nd_jump_root+0xe5/0x160 [11576.197791] ? path_init+0x437/0x520 [11576.199075] path_openat+0xf7/0x500 [11576.200350] ? do_raw_spin_unlock+0x75/0x190 [11576.201653] do_filp_open+0x99/0x140 [11576.202697] ? getname_flags+0x6e/0x330 [11576.203983] ? __check_object_size+0xff/0x256 [11576.205427] ? do_raw_spin_unlock+0x75/0x190 [11576.206944] ? _raw_spin_unlock+0x12/0x30 [11576.208165] do_sys_openat2+0x2b4/0x410 [11576.213043] do_sys_open+0x73/0xa0 [11576.218450] __x64_sys_openat+0x24/0x30 [11576.220201] do_syscall_64+0xc1/0x440 [11576.224746] entry_SYSCALL_64_after_hwframe+0x49/0xae [11576.226511] RIP: 0033:0x7f1a63983332 [11576.231000] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11576.238407] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11576.241697] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11576.244237] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11576.246477] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11576.248931] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11576.251400] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11580.056024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11580.057026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11580.058653] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11580.058666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11580.058750] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11580.058757] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11580.058761] RIP: 0010:native_safe_halt+0x2e/0x50 [11580.058775] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11580.058780] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11580.058785] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11580.058787] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11580.058790] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11580.058792] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11580.058795] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11580.058798] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11580.058802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11580.058804] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11580.058814] Call Trace: [11580.058819] [11580.058825] ? show_regs.cold.9+0x22/0x2f [11580.058849] ? watchdog_timer_fn.cold.6+0x62/0x10a [11580.058856] ? __hrtimer_run_queues+0x1c2/0x4d0 [11580.058861] ? watchdog+0x40/0x40 [11580.058866] ? hrtimer_interrupt+0x128/0x2c0 [11580.058871] ? smp_apic_timer_interrupt+0xa6/0x270 [11580.058878] ? apic_timer_interrupt+0xf/0x20 [11580.058883] [11580.058886] ? native_safe_halt+0x2e/0x50 [11580.058890] kvm_wait+0xcd/0xe0 [11580.058898] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11580.058907] do_raw_spin_lock+0x177/0x190 [11580.058912] _raw_spin_lock+0x21/0x30 [11580.058917] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11580.059410] obd_mod_free+0x46/0x140 [ptlrpc] [11580.059643] mdc_commit_open+0x91/0xb0 [mdc] [11580.059680] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11580.059887] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11580.060602] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11580.060688] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11580.060694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11580.060697] RIP: 0010:native_safe_halt+0x2e/0x50 [11580.060707] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11580.060715] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11580.060719] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11580.060722] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11580.060724] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11580.060725] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11580.060728] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11580.060731] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11580.060733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11580.060735] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11580.060743] Call Trace: [11580.060746] [11580.060749] ? show_regs.cold.9+0x22/0x2f [11580.060758] ? watchdog_timer_fn.cold.6+0x62/0x10a [11580.060765] ? __hrtimer_run_queues+0x1c2/0x4d0 [11580.060768] ? watchdog+0x40/0x40 [11580.060772] ? hrtimer_interrupt+0x128/0x2c0 [11580.060775] ? smp_apic_timer_interrupt+0xa6/0x270 [11580.060784] ? apic_timer_interrupt+0xf/0x20 [11580.060788] [11580.060797] ? native_safe_halt+0x2e/0x50 [11580.060802] kvm_wait+0xcd/0xe0 [11580.060808] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11580.060815] do_raw_spin_lock+0x177/0x190 [11580.060819] _raw_spin_lock+0x21/0x30 [11580.060824] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11580.061247] ? do_raw_spin_unlock+0x75/0x190 [11580.061235] after_reply+0xe2c/0x1780 [ptlrpc] [11580.061327] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11580.061567] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11580.061921] ? __next_timer_interrupt+0x160/0x160 [11580.062040] ? do_wait_intr+0xf0/0xf0 [11580.062049] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11580.062322] mdc_close+0x53b/0xd70 [mdc] [11580.062358] lmv_close+0x18d/0x490 [lmv] [11580.061934] ptlrpcd+0xa82/0xe40 [ptlrpc] [11580.062571] ? do_wait_intr+0xf0/0xf0 [11580.062586] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11580.062419] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11580.062971] kthread+0x1d1/0x200 [11580.062976] ? set_kthread_struct+0x70/0x70 [11580.062979] ret_from_fork+0x1f/0x30 [11580.062969] ll_md_real_close+0xa5/0x2a0 [lustre] [11580.063055] ll_md_close+0x274/0x800 [lustre] [11580.063146] ll_file_release+0x128/0xc50 [lustre] [11580.063233] __fput+0x10d/0x440 [11580.063242] ____fput+0x12/0x20 [11580.063251] task_work_run+0xc6/0x110 [11580.063258] exit_to_usermode_loop+0x1dd/0x1f0 [11580.063266] do_syscall_64+0x3d6/0x440 [11580.063270] entry_SYSCALL_64_after_hwframe+0x49/0xae [11580.063276] RIP: 0033:0x7f1a63983cd8 [11580.063283] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11580.063292] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11580.063300] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11580.063303] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11580.063305] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11580.063307] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11580.063309] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11580.510925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11580.540800] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11580.551685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11580.557242] RIP: 0010:native_safe_halt+0x2e/0x50 [11580.560347] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11580.572195] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11580.581215] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11580.585102] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [11580.588129] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11580.590675] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11580.594963] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11580.598936] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11580.606123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11580.609704] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11580.614273] Call Trace: [11580.618426] [11580.619535] ? show_regs.cold.9+0x22/0x2f [11580.623472] ? watchdog_timer_fn.cold.6+0x62/0x10a [11580.632116] ? __hrtimer_run_queues+0x1c2/0x4d0 [11580.634658] ? watchdog+0x40/0x40 [11580.635861] ? hrtimer_interrupt+0x128/0x2c0 [11580.637547] ? smp_apic_timer_interrupt+0xa6/0x270 [11580.639310] ? apic_timer_interrupt+0xf/0x20 [11580.646486] [11580.647191] ? native_safe_halt+0x2e/0x50 [11580.648692] kvm_wait+0xcd/0xe0 [11580.650412] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11580.652912] do_raw_spin_lock+0x177/0x190 [11580.656466] _raw_spin_lock+0x21/0x30 [11580.657963] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11580.660985] ? do_raw_spin_unlock+0x75/0x190 [11580.663668] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11580.667417] ? do_wait_intr+0xf0/0xf0 [11580.669255] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11580.673952] mdc_close+0x53b/0xd70 [mdc] [11580.676306] lmv_close+0x18d/0x490 [lmv] [11580.678261] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11580.682484] ll_md_real_close+0xa5/0x2a0 [lustre] [11580.686928] ll_md_close+0x274/0x800 [lustre] [11580.688656] ll_file_release+0x128/0xc50 [lustre] [11580.693689] __fput+0x10d/0x440 [11580.696282] ____fput+0x12/0x20 [11580.703327] task_work_run+0xc6/0x110 [11580.712373] exit_to_usermode_loop+0x1dd/0x1f0 [11580.716288] do_syscall_64+0x3d6/0x440 [11580.718569] entry_SYSCALL_64_after_hwframe+0x49/0xae [11580.731430] RIP: 0033:0x7f1a63983cd8 [11580.733306] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11580.749155] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11580.753285] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11580.760113] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11580.770231] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11580.773210] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11580.775366] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11581.922132] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9558s! [11581.932972] Showing busy workqueues and worker pools: [11581.939380] workqueue events: flags=0x0 [11581.941693] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11581.946630] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11581.950415] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11581.953221] pending: sec_gc_main [ptlrpc] [11581.956518] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11581.961023] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11581.966682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11581.972706] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11581.978461] workqueue events_long: flags=0x0 [11581.980300] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11581.983817] pending: lu_objects_flush [obdclass] [11581.986901] workqueue events_unbound: flags=0x2 [11581.989546] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11581.992831] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11581.998562] workqueue events_power_efficient: flags=0x80 [11582.001413] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11582.005333] pending: neigh_periodic_work, do_cache_clean [11582.009077] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11582.013361] pending: neigh_periodic_work, check_lifetime [11582.021201] workqueue mm_percpu_wq: flags=0x8 [11582.023101] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11582.029265] pending: vmstat_update [11582.035734] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11582.041617] pending: vmstat_update [11582.044249] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11582.047690] pending: vmstat_update [11582.049772] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11582.052810] pending: vmstat_update [11582.054533] workqueue writeback: flags=0x4a [11582.056749] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11582.062305] pending: wb_workfn, wb_workfn [11582.072437] workqueue xprtiod: flags=0xa [11582.073917] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11582.082610] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11582.090258] workqueue ipv6_addrconf: flags=0x40008 [11582.091400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11582.096383] pending: addrconf_verify_work [11582.099573] workqueue knbd0-recv: flags=0x1a [11582.102279] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11582.112642] in-flight: 647:recv_work [11582.114067] workqueue ptlrpc_pinger: flags=0x2 [11582.118606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11582.122508] pending: ptlrpc_pinger_main [ptlrpc] [11582.125556] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11572s workers=2 idle: 652 [11608.053709] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11608.056029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11608.057017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11608.057024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11608.057089] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11608.057093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11608.057096] RIP: 0010:native_safe_halt+0x2e/0x50 [11608.057107] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11608.057110] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11608.057113] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11608.057115] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [11608.057117] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11608.057118] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11608.057120] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11608.057122] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11608.057124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11608.057126] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11608.057132] Call Trace: [11608.057137] [11608.057141] ? show_regs.cold.9+0x22/0x2f [11608.057149] ? watchdog_timer_fn.cold.6+0x62/0x10a [11608.057154] ? __hrtimer_run_queues+0x1c2/0x4d0 [11608.057156] ? watchdog+0x40/0x40 [11608.057160] ? hrtimer_interrupt+0x128/0x2c0 [11608.057164] ? smp_apic_timer_interrupt+0xa6/0x270 [11608.057171] ? apic_timer_interrupt+0xf/0x20 [11608.057176] [11608.057179] ? native_safe_halt+0x2e/0x50 [11608.057184] kvm_wait+0xcd/0xe0 [11608.057192] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11608.057213] do_raw_spin_lock+0x177/0x190 [11608.057217] _raw_spin_lock+0x21/0x30 [11608.057222] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11608.057517] ? do_raw_spin_unlock+0x75/0x190 [11608.057523] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11608.057720] ? do_wait_intr+0xf0/0xf0 [11608.057729] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11608.057939] mdc_close+0x53b/0xd70 [mdc] [11608.057978] lmv_close+0x18d/0x490 [lmv] [11608.058004] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11608.058108] ll_md_real_close+0xa5/0x2a0 [lustre] [11608.058167] ll_md_close+0x274/0x800 [lustre] [11608.058254] ll_file_release+0x128/0xc50 [lustre] [11608.058352] __fput+0x10d/0x440 [11608.058360] ____fput+0x12/0x20 [11608.058365] task_work_run+0xc6/0x110 [11608.058373] exit_to_usermode_loop+0x1dd/0x1f0 [11608.058379] do_syscall_64+0x3d6/0x440 [11608.058385] entry_SYSCALL_64_after_hwframe+0x49/0xae [11608.058393] RIP: 0033:0x7f1a63983cd8 [11608.058399] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11608.058403] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11608.058407] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11608.058409] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11608.058411] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11608.058413] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11608.058416] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11608.058522] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11608.058528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11608.058589] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11608.058593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11608.058595] RIP: 0010:native_safe_halt+0x2e/0x50 [11608.058602] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11608.058605] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11608.058609] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11608.058612] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11608.058614] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11608.058616] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11608.058619] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11608.058621] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11608.058624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11608.058626] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11608.058632] Call Trace: [11608.058635] [11608.058637] ? show_regs.cold.9+0x22/0x2f [11608.058645] ? watchdog_timer_fn.cold.6+0x62/0x10a [11608.058650] ? __hrtimer_run_queues+0x1c2/0x4d0 [11608.058654] ? watchdog+0x40/0x40 [11608.058658] ? hrtimer_interrupt+0x128/0x2c0 [11608.058661] ? smp_apic_timer_interrupt+0xa6/0x270 [11608.058667] ? apic_timer_interrupt+0xf/0x20 [11608.058671] [11608.058674] ? native_safe_halt+0x2e/0x50 [11608.058678] kvm_wait+0xcd/0xe0 [11608.058684] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11608.058689] do_raw_spin_lock+0x177/0x190 [11608.058693] _raw_spin_lock+0x21/0x30 [11608.058697] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11608.058940] obd_mod_free+0x46/0x140 [ptlrpc] [11608.059130] mdc_commit_open+0x91/0xb0 [mdc] [11608.059161] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11608.059374] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11608.059577] after_reply+0xe2c/0x1780 [ptlrpc] [11608.059778] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11608.059971] ? __next_timer_interrupt+0x160/0x160 [11608.059980] ptlrpcd+0xa82/0xe40 [ptlrpc] [11608.060194] ? do_wait_intr+0xf0/0xf0 [11608.060201] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11608.060411] kthread+0x1d1/0x200 [11608.060417] ? set_kthread_struct+0x70/0x70 [11608.060421] ret_from_fork+0x1f/0x30 [11608.060836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11608.060909] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11608.060915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11608.060918] RIP: 0010:native_safe_halt+0x2e/0x50 [11608.060927] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11608.060931] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11608.060935] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11608.060938] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [11608.060940] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11608.060942] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11608.060944] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11608.060947] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11608.060950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11608.060953] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11608.060964] Call Trace: [11608.060968] [11608.060972] ? show_regs.cold.9+0x22/0x2f [11608.060981] ? watchdog_timer_fn.cold.6+0x62/0x10a [11608.060986] ? __hrtimer_run_queues+0x1c2/0x4d0 [11608.060991] ? watchdog+0x40/0x40 [11608.060995] ? hrtimer_interrupt+0x128/0x2c0 [11608.060999] ? smp_apic_timer_interrupt+0xa6/0x270 [11608.061005] ? apic_timer_interrupt+0xf/0x20 [11608.061009] [11608.061012] ? native_safe_halt+0x2e/0x50 [11608.061016] kvm_wait+0xcd/0xe0 [11608.061023] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11608.061031] do_raw_spin_lock+0x177/0x190 [11608.061036] _raw_spin_lock+0x21/0x30 [11608.061041] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11608.061268] ? do_raw_spin_unlock+0x75/0x190 [11608.061274] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11608.064721] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11608.064806] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11608.064814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11608.064817] RIP: 0010:native_safe_halt+0x2e/0x50 [11608.064832] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11608.064837] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11608.064842] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11608.064844] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11608.064846] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11608.064848] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11608.064851] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11608.064855] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11608.064858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11608.064860] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11608.064869] Call Trace: [11608.064875] [11608.064881] ? show_regs.cold.9+0x22/0x2f [11608.064892] ? watchdog_timer_fn.cold.6+0x62/0x10a [11608.064899] ? __hrtimer_run_queues+0x1c2/0x4d0 [11608.064904] ? watchdog+0x40/0x40 [11608.064909] ? hrtimer_interrupt+0x128/0x2c0 [11608.064913] ? smp_apic_timer_interrupt+0xa6/0x270 [11608.064920] ? apic_timer_interrupt+0xf/0x20 [11608.064925] [11608.064927] ? native_safe_halt+0x2e/0x50 [11608.064931] kvm_wait+0xcd/0xe0 [11608.064939] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11608.064947] do_raw_spin_lock+0x177/0x190 [11608.064952] _raw_spin_lock+0x21/0x30 [11608.064956] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11608.065862] ? do_raw_spin_unlock+0x75/0x190 [11608.065869] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11608.066123] ? do_wait_intr+0xf0/0xf0 [11608.066136] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11608.066398] mdc_close+0x53b/0xd70 [mdc] [11608.066466] lmv_close+0x18d/0x490 [lmv] [11608.066506] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11608.066640] ll_md_real_close+0xa5/0x2a0 [lustre] [11608.066738] ll_md_close+0x274/0x800 [lustre] [11608.066839] ll_file_release+0x128/0xc50 [lustre] [11608.067243] __fput+0x10d/0x440 [11608.067252] ____fput+0x12/0x20 [11608.067256] task_work_run+0xc6/0x110 [11608.067265] exit_to_usermode_loop+0x1dd/0x1f0 [11608.067273] do_syscall_64+0x3d6/0x440 [11608.067278] entry_SYSCALL_64_after_hwframe+0x49/0xae [11608.067285] RIP: 0033:0x7f1a63983cd8 [11608.067292] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11608.067296] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11608.067301] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11608.067303] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11608.067305] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11608.067308] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11608.067310] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11609.078567] ? do_wait_intr+0xf0/0xf0 [11609.082307] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11609.090556] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11609.095465] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11609.101547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11609.104586] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11609.110600] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11609.114552] mdc_intent_lock+0x353/0x7e0 [mdc] [11609.119206] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11609.124623] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11609.129800] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11609.139472] lmv_intent_open+0x338/0x1150 [lmv] [11609.146482] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11609.153922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11609.160136] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11609.161875] ll_intent_lock+0xc5/0xaa0 [lustre] [11609.174030] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11609.175874] ll_lookup_it+0xe0e/0x2270 [lustre] [11609.181234] ? do_raw_spin_unlock+0x75/0x190 [11609.184169] ? _raw_spin_unlock+0x12/0x30 [11609.188751] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11609.195502] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11609.199422] ll_atomic_open+0x36a/0xd70 [lustre] [11609.206858] lookup_open+0x356/0xb40 [11609.209084] do_last+0x313/0xfc0 [11609.218191] ? nd_jump_root+0xe5/0x160 [11609.221867] ? path_init+0x437/0x520 [11609.225740] path_openat+0xf7/0x500 [11609.237453] ? do_raw_spin_unlock+0x75/0x190 [11609.248599] do_filp_open+0x99/0x140 [11609.249830] ? getname_flags+0x6e/0x330 [11609.260423] ? __check_object_size+0xff/0x256 [11609.268587] ? do_raw_spin_unlock+0x75/0x190 [11609.273949] ? _raw_spin_unlock+0x12/0x30 [11609.281839] do_sys_openat2+0x2b4/0x410 [11609.285493] do_sys_open+0x73/0xa0 [11609.286727] __x64_sys_openat+0x24/0x30 [11609.294489] do_syscall_64+0xc1/0x440 [11609.296571] entry_SYSCALL_64_after_hwframe+0x49/0xae [11609.301751] RIP: 0033:0x7f1a63983332 [11609.306221] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11609.322071] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11609.331879] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11609.336986] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11609.342691] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11609.349804] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11609.356750] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11612.641037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9589s! [11612.644371] Showing busy workqueues and worker pools: [11612.650101] workqueue events: flags=0x0 [11612.655446] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11612.660533] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11612.668608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11612.681401] pending: sec_gc_main [ptlrpc] [11612.686056] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11612.697508] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11612.709735] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11612.713709] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11612.721945] workqueue events_long: flags=0x0 [11612.726646] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11612.732658] pending: lu_objects_flush [obdclass] [11612.734731] workqueue events_unbound: flags=0x2 [11612.742678] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11612.748982] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11612.762867] workqueue events_power_efficient: flags=0x80 [11612.766695] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11612.778626] pending: neigh_periodic_work, do_cache_clean [11612.782554] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11612.790932] pending: neigh_periodic_work, check_lifetime [11612.794753] workqueue mm_percpu_wq: flags=0x8 [11612.801978] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11612.806250] pending: vmstat_update [11612.809586] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11612.817745] pending: vmstat_update [11612.820576] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11612.823710] pending: vmstat_update [11612.825157] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11612.832274] pending: vmstat_update [11612.837944] workqueue writeback: flags=0x4a [11612.843718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11612.851703] pending: wb_workfn, wb_workfn [11612.853459] workqueue xprtiod: flags=0xa [11612.859832] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11612.863173] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11612.867183] workqueue ipv6_addrconf: flags=0x40008 [11612.869561] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11612.872638] pending: addrconf_verify_work [11612.874883] workqueue knbd0-recv: flags=0x1a [11612.876693] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11612.879888] in-flight: 647:recv_work [11612.881707] workqueue ptlrpc_pinger: flags=0x2 [11612.883904] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11612.886575] pending: ptlrpc_pinger_main [ptlrpc] [11612.889235] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11602s workers=2 idle: 652 [11622.662801] rcu: INFO: rcu_sched self-detected stall on CPU [11622.668027] rcu: 3-...!: (6033015 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [11622.684526] (t=9600278 jiffies g=269633 q=3440) [11622.692999] rcu: rcu_sched kthread starved for 9599932 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [11622.706930] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11622.724337] rcu: RCU grace-period kthread stack dump: [11622.729025] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11622.737787] Call Trace: [11622.739648] __schedule+0x351/0xcb0 [11622.745794] schedule+0xc0/0x180 [11622.748611] schedule_timeout+0xb4/0x190 [11622.749989] ? do_raw_spin_unlock+0x75/0x190 [11622.754529] ? __next_timer_interrupt+0x160/0x160 [11622.756625] ? prepare_to_swait_event+0xf3/0x140 [11622.760507] rcu_gp_kthread+0x8dd/0x15b0 [11622.762137] ? force_qs_rnp+0x2c0/0x2c0 [11622.763924] kthread+0x1d1/0x200 [11622.765934] ? set_kthread_struct+0x70/0x70 [11622.774417] ret_from_fork+0x1f/0x30 [11622.788825] rcu: Stack dump where RCU GP kthread last ran: [11622.790592] Sending NMI from CPU 3 to CPUs 0: [11622.814691] NMI backtrace for cpu 0 [11622.814699] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11622.814701] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11622.814703] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11622.814704] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11622.814706] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 [11622.814709] RAX: 0000000000007f85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11622.814710] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11622.814712] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11622.814713] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11622.814715] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11622.814716] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11622.814718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11622.814719] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11622.814720] Call Trace: [11622.814721] [11622.814722] ? show_regs.cold.9+0x22/0x2f [11622.814723] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11622.814724] ? nmi_cpu_backtrace_handler+0x15/0x40 [11622.814725] ? nmi_handle+0xbd/0x210 [11622.814726] ? default_do_nmi+0x4b/0x1b0 [11622.814727] ? do_nmi+0x280/0x380 [11622.814728] ? end_repeat_nmi+0x16/0x1f [11622.814729] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11622.814730] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11622.814731] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11622.814732] [11622.814733] do_raw_spin_lock+0x177/0x190 [11622.814734] _raw_spin_lock+0x21/0x30 [11622.814735] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11622.814736] ? do_raw_spin_unlock+0x75/0x190 [11622.814737] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11622.814738] ? do_wait_intr+0xf0/0xf0 [11622.814739] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11622.814740] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11622.814741] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11622.814742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11622.814744] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11622.814745] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11622.814746] mdc_intent_lock+0x353/0x7e0 [mdc] [11622.814747] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11622.814748] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11622.814749] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11622.814750] lmv_intent_open+0x338/0x1150 [lmv] [11622.814751] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11622.814752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11622.814753] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11622.814754] ll_intent_lock+0xc5/0xaa0 [lustre] [11622.814755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11622.814756] ll_lookup_it+0xe0e/0x2270 [lustre] [11622.814757] ? do_raw_spin_unlock+0x75/0x190 [11622.814758] ? _raw_spin_unlock+0x12/0x30 [11622.814759] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11622.814760] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11622.814761] ll_atomic_open+0x36a/0xd70 [lustre] [11622.814762] lookup_open+0x356/0xb40 [11622.814763] do_last+0x313/0xfc0 [11622.814764] ? nd_jump_root+0xe5/0x160 [11622.814765] ? path_init+0x437/0x520 [11622.814767] path_openat+0xf7/0x500 [11622.814768] ? do_raw_spin_unlock+0x75/0x190 [11622.814769] do_filp_open+0x99/0x140 [11622.814770] ? getname_flags+0x6e/0x330 [11622.814771] ? __check_object_size+0xff/0x256 [11622.814772] ? do_raw_spin_unlock+0x75/0x190 [11622.814773] ? _raw_spin_unlock+0x12/0x30 [11622.814774] do_sys_openat2+0x2b4/0x410 [11622.814776] do_sys_open+0x73/0xa0 [11622.814776] __x64_sys_openat+0x24/0x30 [11622.814778] do_syscall_64+0xc1/0x440 [11622.814779] entry_SYSCALL_64_after_hwframe+0x49/0xae [11622.814780] RIP: 0033:0x7f1a63983332 [11622.814782] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11622.814784] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11622.814787] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11622.814788] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11622.814790] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11622.814791] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11622.814792] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11622.820313] NMI backtrace for cpu 3 [11623.428095] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11623.437729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11623.445493] Call Trace: [11623.446990] [11623.451450] ? dump_stack+0xbb/0x10e [11623.453258] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11623.459161] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11623.468581] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11623.473463] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11623.480487] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11623.486808] ? rcu_dump_cpu_stacks+0x11b/0x171 [11623.490032] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11623.495538] ? update_process_times+0x6d/0xc0 [11623.500909] ? tick_sched_handle.isra.18+0x27/0x90 [11623.502585] ? tick_sched_timer+0x53/0xb0 [11623.508024] ? __hrtimer_run_queues+0x1c2/0x4d0 [11623.509696] ? tick_sched_handle.isra.18+0x90/0x90 [11623.515857] ? hrtimer_interrupt+0x128/0x2c0 [11623.518999] ? smp_apic_timer_interrupt+0xa6/0x270 [11623.523862] ? apic_timer_interrupt+0xf/0x20 [11623.529252] [11623.530250] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11623.539457] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11623.542954] ? do_raw_spin_lock+0x177/0x190 [11623.552633] ? _raw_spin_lock+0x21/0x30 [11623.555845] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11623.562931] ? obd_mod_free+0x46/0x140 [ptlrpc] [11623.569004] ? mdc_commit_open+0x91/0xb0 [mdc] [11623.574421] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11623.582537] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11623.592001] ? after_reply+0xe2c/0x1780 [ptlrpc] [11623.600206] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11623.606843] ? __next_timer_interrupt+0x160/0x160 [11623.613525] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11623.622128] ? do_wait_intr+0xf0/0xf0 [11623.627683] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11623.633123] ? kthread+0x1d1/0x200 [11623.635143] ? set_kthread_struct+0x70/0x70 [11623.637434] ? ret_from_fork+0x1f/0x30 [11636.056031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11636.058699] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11636.060296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11636.060369] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11636.060377] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11636.060381] RIP: 0010:native_safe_halt+0x2e/0x50 [11636.060394] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11636.060398] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11636.060402] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11636.060405] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11636.060407] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11636.060409] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11636.060411] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11636.060415] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11636.060418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11636.060420] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11636.060433] Call Trace: [11636.060438] [11636.060444] ? show_regs.cold.9+0x22/0x2f [11636.060459] ? watchdog_timer_fn.cold.6+0x62/0x10a [11636.060466] ? __hrtimer_run_queues+0x1c2/0x4d0 [11636.060471] ? watchdog+0x40/0x40 [11636.060475] ? hrtimer_interrupt+0x128/0x2c0 [11636.060479] ? smp_apic_timer_interrupt+0xa6/0x270 [11636.060487] ? apic_timer_interrupt+0xf/0x20 [11636.060491] [11636.060494] ? native_safe_halt+0x2e/0x50 [11636.060498] kvm_wait+0xcd/0xe0 [11636.060506] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11636.060515] do_raw_spin_lock+0x177/0x190 [11636.060519] _raw_spin_lock+0x21/0x30 [11636.060523] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11636.060946] ? do_raw_spin_unlock+0x75/0x190 [11636.060952] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11636.061304] ? do_wait_intr+0xf0/0xf0 [11636.061313] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11636.061514] mdc_close+0x53b/0xd70 [mdc] [11636.061555] lmv_close+0x18d/0x490 [lmv] [11636.061586] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11636.061771] ll_md_real_close+0xa5/0x2a0 [lustre] [11636.061857] ll_md_close+0x274/0x800 [lustre] [11636.061950] ll_file_release+0x128/0xc50 [lustre] [11636.062030] __fput+0x10d/0x440 [11636.062040] ____fput+0x12/0x20 [11636.062044] task_work_run+0xc6/0x110 [11636.062051] exit_to_usermode_loop+0x1dd/0x1f0 [11636.062096] do_syscall_64+0x3d6/0x440 [11636.062101] entry_SYSCALL_64_after_hwframe+0x49/0xae [11636.062112] RIP: 0033:0x7f1a63983cd8 [11636.062119] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11636.062123] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11636.062127] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11636.062129] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11636.062131] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11636.062134] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11636.062136] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11636.232079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11636.266817] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11636.277856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11636.286350] RIP: 0010:native_safe_halt+0x2e/0x50 [11636.301332] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11636.331131] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11636.337645] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11636.340493] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [11636.343892] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11636.347294] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11636.353778] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11636.356840] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11636.359678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11636.362776] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11636.368428] Call Trace: [11636.370499] [11636.371614] ? show_regs.cold.9+0x22/0x2f [11636.372732] ? watchdog_timer_fn.cold.6+0x62/0x10a [11636.375495] ? __hrtimer_run_queues+0x1c2/0x4d0 [11636.378887] ? watchdog+0x40/0x40 [11636.383300] ? hrtimer_interrupt+0x128/0x2c0 [11636.386642] ? smp_apic_timer_interrupt+0xa6/0x270 [11636.388393] ? apic_timer_interrupt+0xf/0x20 [11636.400759] [11636.407424] ? native_safe_halt+0x2e/0x50 [11636.413361] kvm_wait+0xcd/0xe0 [11636.416648] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11636.420384] do_raw_spin_lock+0x177/0x190 [11636.423753] _raw_spin_lock+0x21/0x30 [11636.425624] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11636.428778] ? do_raw_spin_unlock+0x75/0x190 [11636.431671] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11636.435747] ? do_wait_intr+0xf0/0xf0 [11636.438067] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11636.443513] mdc_close+0x53b/0xd70 [mdc] [11636.455728] lmv_close+0x18d/0x490 [lmv] [11636.458650] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11636.464784] ll_md_real_close+0xa5/0x2a0 [lustre] [11636.475409] ll_md_close+0x274/0x800 [lustre] [11636.478419] ll_file_release+0x128/0xc50 [lustre] [11636.482713] __fput+0x10d/0x440 [11636.492353] ____fput+0x12/0x20 [11636.500898] task_work_run+0xc6/0x110 [11636.506622] exit_to_usermode_loop+0x1dd/0x1f0 [11636.519047] do_syscall_64+0x3d6/0x440 [11636.520474] entry_SYSCALL_64_after_hwframe+0x49/0xae [11636.534027] RIP: 0033:0x7f1a63983cd8 [11636.541843] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11636.565466] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11636.574404] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11636.583325] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11636.585902] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11636.605590] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11636.612200] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11643.363037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9620s! [11643.367017] Showing busy workqueues and worker pools: [11643.369666] workqueue events: flags=0x0 [11643.370788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11643.377658] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11643.387767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11643.391582] pending: sec_gc_main [ptlrpc] [11643.394923] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11643.401093] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11643.414777] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11643.419687] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11643.430655] workqueue events_long: flags=0x0 [11643.436100] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11643.439796] pending: lu_objects_flush [obdclass] [11643.445049] workqueue events_unbound: flags=0x2 [11643.448128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11643.455516] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11643.461982] workqueue events_power_efficient: flags=0x80 [11643.467677] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11643.476350] pending: neigh_periodic_work, do_cache_clean [11643.489414] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11643.499792] pending: neigh_periodic_work, check_lifetime [11643.507948] workqueue mm_percpu_wq: flags=0x8 [11643.512784] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11643.525314] pending: vmstat_update [11643.530955] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11643.539127] pending: vmstat_update [11643.546513] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11643.557566] pending: vmstat_update [11643.560950] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11643.576344] pending: vmstat_update [11643.582836] workqueue writeback: flags=0x4a [11643.589094] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11643.593795] pending: wb_workfn, wb_workfn [11643.599200] workqueue xprtiod: flags=0xa [11643.605662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11643.609626] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11643.619635] workqueue ipv6_addrconf: flags=0x40008 [11643.628680] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11643.636848] pending: addrconf_verify_work [11643.640363] workqueue knbd0-recv: flags=0x1a [11643.648398] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11643.662587] in-flight: 647:recv_work [11643.664089] workqueue ptlrpc_pinger: flags=0x2 [11643.673633] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11643.680586] pending: ptlrpc_pinger_main [ptlrpc] [11643.689375] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11633s workers=2 idle: 652 [11648.058064] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [11648.061810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11648.089049] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11648.093396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11648.097228] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11648.101207] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11648.110845] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11648.114899] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11648.118578] RDX: 000000000000512c RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11648.122444] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11648.125492] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11648.132463] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11648.137212] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11648.141903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11648.145130] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11648.148912] Call Trace: [11648.151080] [11648.152091] ? show_regs.cold.9+0x22/0x2f [11648.153895] ? watchdog_timer_fn.cold.6+0x62/0x10a [11648.155800] ? __hrtimer_run_queues+0x1c2/0x4d0 [11648.158158] ? watchdog+0x40/0x40 [11648.160495] ? hrtimer_interrupt+0x128/0x2c0 [11648.162553] ? smp_apic_timer_interrupt+0xa6/0x270 [11648.165255] ? apic_timer_interrupt+0xf/0x20 [11648.169814] [11648.170622] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11648.174359] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11648.177815] do_raw_spin_lock+0x177/0x190 [11648.180053] _raw_spin_lock+0x21/0x30 [11648.183618] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11648.186633] obd_mod_free+0x46/0x140 [ptlrpc] [11648.189673] mdc_commit_open+0x91/0xb0 [mdc] [11648.192138] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11648.195604] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11648.198886] after_reply+0xe2c/0x1780 [ptlrpc] [11648.201882] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11648.205230] ? __next_timer_interrupt+0x160/0x160 [11648.207478] ptlrpcd+0xa82/0xe40 [ptlrpc] [11648.210317] ? do_wait_intr+0xf0/0xf0 [11648.212041] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11648.215022] kthread+0x1d1/0x200 [11648.216490] ? set_kthread_struct+0x70/0x70 [11648.218660] ret_from_fork+0x1f/0x30 [11664.056125] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11664.059835] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11664.063496] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11664.063573] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11664.063579] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11664.063582] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11664.063595] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11664.063600] RSP: 0018:ffffc38542073af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [11664.063604] RAX: 0000000000004f88 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11664.063607] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11664.063609] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11664.063611] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11664.063613] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11664.063616] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11664.063619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11664.063621] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11664.063629] Call Trace: [11664.063635] [11664.063640] ? show_regs.cold.9+0x22/0x2f [11664.063648] ? watchdog_timer_fn.cold.6+0x62/0x10a [11664.063652] ? __hrtimer_run_queues+0x1c2/0x4d0 [11664.063655] ? watchdog+0x40/0x40 [11664.063658] ? hrtimer_interrupt+0x128/0x2c0 [11664.063660] ? smp_apic_timer_interrupt+0xa6/0x270 [11664.063667] ? apic_timer_interrupt+0xf/0x20 [11664.063670] [11664.063672] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11664.063676] do_raw_spin_lock+0x177/0x190 [11664.063680] _raw_spin_lock+0x21/0x30 [11664.063684] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11664.073543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [11664.106566] ? do_raw_spin_unlock+0x75/0x190 [11664.106589] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11664.107471] ? do_wait_intr+0xf0/0xf0 [11664.107481] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11664.107686] mdc_close+0x53b/0xd70 [mdc] [11664.107725] lmv_close+0x18d/0x490 [lmv] [11664.107759] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11664.108089] ll_md_real_close+0xa5/0x2a0 [lustre] [11664.108172] ll_md_close+0x274/0x800 [lustre] [11664.108261] ll_file_release+0x128/0xc50 [lustre] [11664.108347] __fput+0x10d/0x440 [11664.108356] ____fput+0x12/0x20 [11664.108359] task_work_run+0xc6/0x110 [11664.108366] exit_to_usermode_loop+0x1dd/0x1f0 [11664.108374] do_syscall_64+0x3d6/0x440 [11664.108378] entry_SYSCALL_64_after_hwframe+0x49/0xae [11664.108393] RIP: 0033:0x7f1a63983cd8 [11664.108403] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11664.108407] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11664.108411] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11664.108413] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11664.108415] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11664.108417] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11664.108419] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11664.449645] sha512_ssse3 sha512_generic [11664.457579] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11664.467425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11664.474392] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11664.482214] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11664.498263] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11664.506174] RAX: 0000000000003e89 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11664.512033] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11664.515944] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11664.520559] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11664.531628] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11664.544023] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11664.552508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11664.559431] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11664.571646] Call Trace: [11664.574753] [11664.578329] ? show_regs.cold.9+0x22/0x2f [11664.581485] ? watchdog_timer_fn.cold.6+0x62/0x10a [11664.584281] ? __hrtimer_run_queues+0x1c2/0x4d0 [11664.588696] ? watchdog+0x40/0x40 [11664.592496] ? hrtimer_interrupt+0x128/0x2c0 [11664.595630] ? smp_apic_timer_interrupt+0xa6/0x270 [11664.601523] ? apic_timer_interrupt+0xf/0x20 [11664.605293] [11664.607369] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11664.613941] do_raw_spin_lock+0x177/0x190 [11664.615687] _raw_spin_lock+0x21/0x30 [11664.617610] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11664.621589] ? do_raw_spin_unlock+0x75/0x190 [11664.626649] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11664.629589] ? do_wait_intr+0xf0/0xf0 [11664.632062] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11664.634126] mdc_close+0x53b/0xd70 [mdc] [11664.636539] lmv_close+0x18d/0x490 [lmv] [11664.638621] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11664.642493] ll_md_real_close+0xa5/0x2a0 [lustre] [11664.645365] ll_md_close+0x274/0x800 [lustre] [11664.647629] ll_file_release+0x128/0xc50 [lustre] [11664.649841] __fput+0x10d/0x440 [11664.651500] ____fput+0x12/0x20 [11664.652716] task_work_run+0xc6/0x110 [11664.654304] exit_to_usermode_loop+0x1dd/0x1f0 [11664.656109] do_syscall_64+0x3d6/0x440 [11664.657644] entry_SYSCALL_64_after_hwframe+0x49/0xae [11664.660387] RIP: 0033:0x7f1a63983cd8 [11664.662298] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11664.672759] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11664.676975] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11664.681177] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11664.685245] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11664.688657] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11664.692629] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11668.053022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11668.056270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11668.090732] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11668.098240] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11668.107533] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11668.112556] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11668.127602] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11668.133625] RAX: 00000000000037a5 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11668.140196] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11668.147295] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11668.153261] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11668.162115] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11668.170889] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11668.176628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11668.180378] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11668.189551] Call Trace: [11668.192103] [11668.192903] ? show_regs.cold.9+0x22/0x2f [11668.194357] ? watchdog_timer_fn.cold.6+0x62/0x10a [11668.199569] ? __hrtimer_run_queues+0x1c2/0x4d0 [11668.203425] ? watchdog+0x40/0x40 [11668.204926] ? hrtimer_interrupt+0x128/0x2c0 [11668.208408] ? smp_apic_timer_interrupt+0xa6/0x270 [11668.210608] ? apic_timer_interrupt+0xf/0x20 [11668.212450] [11668.215459] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11668.219699] do_raw_spin_lock+0x177/0x190 [11668.221490] _raw_spin_lock+0x21/0x30 [11668.224359] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11668.226737] ? do_raw_spin_unlock+0x75/0x190 [11668.229719] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11668.234124] ? do_wait_intr+0xf0/0xf0 [11668.237157] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11668.238537] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11668.241459] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11668.244331] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11668.245810] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11668.248510] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11668.250004] mdc_intent_lock+0x353/0x7e0 [mdc] [11668.251396] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11668.254521] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11668.257624] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11668.259159] lmv_intent_open+0x338/0x1150 [lmv] [11668.260299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11668.263943] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11668.267228] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11668.268490] ll_intent_lock+0xc5/0xaa0 [lustre] [11668.270174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11668.272626] ll_lookup_it+0xe0e/0x2270 [lustre] [11668.273840] ? do_raw_spin_unlock+0x75/0x190 [11668.275067] ? _raw_spin_unlock+0x12/0x30 [11668.276595] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11668.279191] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11668.280408] ll_atomic_open+0x36a/0xd70 [lustre] [11668.284210] lookup_open+0x356/0xb40 [11668.291363] do_last+0x313/0xfc0 [11668.293394] ? nd_jump_root+0xe5/0x160 [11668.294394] ? path_init+0x437/0x520 [11668.295588] path_openat+0xf7/0x500 [11668.298461] ? do_raw_spin_unlock+0x75/0x190 [11668.301680] do_filp_open+0x99/0x140 [11668.309584] ? getname_flags+0x6e/0x330 [11668.312329] ? __check_object_size+0xff/0x256 [11668.314605] ? do_raw_spin_unlock+0x75/0x190 [11668.317626] ? _raw_spin_unlock+0x12/0x30 [11668.320071] do_sys_openat2+0x2b4/0x410 [11668.323193] do_sys_open+0x73/0xa0 [11668.325134] __x64_sys_openat+0x24/0x30 [11668.328404] do_syscall_64+0xc1/0x440 [11668.330690] entry_SYSCALL_64_after_hwframe+0x49/0xae [11668.332504] RIP: 0033:0x7f1a63983332 [11668.338674] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11668.353244] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11668.357278] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11668.366326] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11668.377412] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11668.381134] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11668.386689] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11674.080043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9651s! [11674.084125] Showing busy workqueues and worker pools: [11674.085650] workqueue events: flags=0x0 [11674.088778] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11674.094568] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11674.099651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11674.107091] pending: sec_gc_main [ptlrpc] [11674.116462] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11674.122037] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11674.135807] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11674.145819] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11674.157414] workqueue events_long: flags=0x0 [11674.160511] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11674.169969] pending: lu_objects_flush [obdclass] [11674.171839] workqueue events_unbound: flags=0x2 [11674.184070] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11674.192855] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11674.202909] workqueue events_power_efficient: flags=0x80 [11674.205589] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11674.208947] pending: neigh_periodic_work, do_cache_clean [11674.212810] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11674.217044] pending: neigh_periodic_work, check_lifetime [11674.220511] workqueue mm_percpu_wq: flags=0x8 [11674.224148] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11674.230556] pending: vmstat_update [11674.232986] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11674.235644] pending: vmstat_update [11674.237122] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11674.239867] pending: vmstat_update [11674.242152] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11674.244614] pending: vmstat_update [11674.245897] workqueue writeback: flags=0x4a [11674.247375] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11674.250354] pending: wb_workfn, wb_workfn [11674.252445] workqueue xprtiod: flags=0xa [11674.254746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11674.257554] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11674.264101] workqueue ipv6_addrconf: flags=0x40008 [11674.268919] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11674.271881] pending: addrconf_verify_work [11674.273315] workqueue knbd0-recv: flags=0x1a [11674.276084] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11674.278389] in-flight: 647:recv_work [11674.279779] workqueue ptlrpc_pinger: flags=0x2 [11674.282900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11674.285420] pending: ptlrpc_pinger_main [ptlrpc] [11674.287519] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11664s workers=2 idle: 652 [11676.058544] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [11676.061317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11676.103482] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11676.112959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11676.119538] RIP: 0010:native_safe_halt+0x2e/0x50 [11676.123344] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11676.133945] RSP: 0018:ffffc38541af7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11676.137049] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11676.140268] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11676.142941] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11676.146243] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11676.149406] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11676.152565] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11676.156319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11676.159016] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11676.162241] Call Trace: [11676.163362] [11676.164285] ? show_regs.cold.9+0x22/0x2f [11676.166259] ? watchdog_timer_fn.cold.6+0x62/0x10a [11676.167812] ? __hrtimer_run_queues+0x1c2/0x4d0 [11676.169262] ? watchdog+0x40/0x40 [11676.170511] ? hrtimer_interrupt+0x128/0x2c0 [11676.172463] ? smp_apic_timer_interrupt+0xa6/0x270 [11676.174744] ? apic_timer_interrupt+0xf/0x20 [11676.176537] [11676.177745] ? native_safe_halt+0x2e/0x50 [11676.179491] kvm_wait+0xcd/0xe0 [11676.181478] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11676.183974] do_raw_spin_lock+0x177/0x190 [11676.186229] _raw_spin_lock+0x21/0x30 [11676.187994] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11676.190758] obd_mod_free+0x46/0x140 [ptlrpc] [11676.193248] mdc_commit_open+0x91/0xb0 [mdc] [11676.197572] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11676.201561] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11676.204837] after_reply+0xe2c/0x1780 [ptlrpc] [11676.208475] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11676.211589] ? __next_timer_interrupt+0x160/0x160 [11676.213211] ptlrpcd+0xa82/0xe40 [ptlrpc] [11676.215055] ? do_wait_intr+0xf0/0xf0 [11676.216441] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11676.218630] kthread+0x1d1/0x200 [11676.219474] ? set_kthread_struct+0x70/0x70 [11676.220465] ret_from_fork+0x1f/0x30 [11692.056872] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [11692.057346] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11692.065904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11692.065976] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11692.065983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11692.065985] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11692.066000] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11692.066004] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11692.066008] RAX: 0000000000006986 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11692.066010] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11692.066012] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11692.066014] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11692.066016] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11692.066019] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11692.066021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11692.066024] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11692.066034] Call Trace: [11692.066040] [11692.066045] ? show_regs.cold.9+0x22/0x2f [11692.066054] ? watchdog_timer_fn.cold.6+0x62/0x10a [11692.067156] ? __hrtimer_run_queues+0x1c2/0x4d0 [11692.067164] ? watchdog+0x40/0x40 [11692.067168] ? hrtimer_interrupt+0x128/0x2c0 [11692.067172] ? smp_apic_timer_interrupt+0xa6/0x270 [11692.067180] ? apic_timer_interrupt+0xf/0x20 [11692.067185] [11692.067188] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11692.067196] do_raw_spin_lock+0x177/0x190 [11692.067252] _raw_spin_lock+0x21/0x30 [11692.067259] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11692.067765] ? do_raw_spin_unlock+0x75/0x190 [11692.067771] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11692.067950] ? do_wait_intr+0xf0/0xf0 [11692.067957] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11692.069331] mdc_close+0x53b/0xd70 [mdc] [11692.069370] lmv_close+0x18d/0x490 [lmv] [11692.069479] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11692.069724] ll_md_real_close+0xa5/0x2a0 [lustre] [11692.069804] ll_md_close+0x274/0x800 [lustre] [11692.069884] ll_file_release+0x128/0xc50 [lustre] [11692.069965] __fput+0x10d/0x440 [11692.069976] ____fput+0x12/0x20 [11692.069982] task_work_run+0xc6/0x110 [11692.069990] exit_to_usermode_loop+0x1dd/0x1f0 [11692.069998] do_syscall_64+0x3d6/0x440 [11692.070002] entry_SYSCALL_64_after_hwframe+0x49/0xae [11692.070010] RIP: 0033:0x7f1a63983cd8 [11692.070017] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11692.070021] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11692.070026] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11692.070028] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11692.070031] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11692.070033] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11692.070035] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11692.405385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11692.446625] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11692.458647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11692.472311] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11692.484542] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11692.504820] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11692.518036] RAX: 000000000000257a RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11692.523745] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11692.532838] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11692.540178] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11692.552135] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11692.559185] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11692.565985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11692.570200] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11692.575178] Call Trace: [11692.576957] [11692.578346] ? show_regs.cold.9+0x22/0x2f [11692.582178] ? watchdog_timer_fn.cold.6+0x62/0x10a [11692.588029] ? __hrtimer_run_queues+0x1c2/0x4d0 [11692.590683] ? watchdog+0x40/0x40 [11692.593047] ? hrtimer_interrupt+0x128/0x2c0 [11692.596532] ? smp_apic_timer_interrupt+0xa6/0x270 [11692.598324] ? apic_timer_interrupt+0xf/0x20 [11692.603560] [11692.605116] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11692.608671] do_raw_spin_lock+0x177/0x190 [11692.611295] _raw_spin_lock+0x21/0x30 [11692.614389] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11692.618891] ? do_raw_spin_unlock+0x75/0x190 [11692.621624] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11692.626595] ? do_wait_intr+0xf0/0xf0 [11692.629025] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11692.634668] mdc_close+0x53b/0xd70 [mdc] [11692.640163] lmv_close+0x18d/0x490 [lmv] [11692.652344] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11692.666672] ll_md_real_close+0xa5/0x2a0 [lustre] [11692.679618] ll_md_close+0x274/0x800 [lustre] [11692.693901] ll_file_release+0x128/0xc50 [lustre] [11692.700909] __fput+0x10d/0x440 [11692.703501] ____fput+0x12/0x20 [11692.711160] task_work_run+0xc6/0x110 [11692.716392] exit_to_usermode_loop+0x1dd/0x1f0 [11692.721164] do_syscall_64+0x3d6/0x440 [11692.722549] entry_SYSCALL_64_after_hwframe+0x49/0xae [11692.731884] RIP: 0033:0x7f1a63983cd8 [11692.739086] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11692.753593] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11692.760390] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11692.764763] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11692.776385] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11692.784657] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11692.799136] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11700.053080] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11700.062703] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11700.085931] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11700.091204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11700.095697] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11700.101694] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11700.113604] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11700.117587] RAX: 0000000000005117 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11700.120989] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11700.124325] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11700.127753] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11700.131553] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11700.135143] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11700.139389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11700.142147] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11700.145330] Call Trace: [11700.146580] [11700.147611] ? show_regs.cold.9+0x22/0x2f [11700.149399] ? watchdog_timer_fn.cold.6+0x62/0x10a [11700.151723] ? __hrtimer_run_queues+0x1c2/0x4d0 [11700.153605] ? watchdog+0x40/0x40 [11700.154973] ? hrtimer_interrupt+0x128/0x2c0 [11700.157028] ? smp_apic_timer_interrupt+0xa6/0x270 [11700.159275] ? apic_timer_interrupt+0xf/0x20 [11700.161181] [11700.162245] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11700.164646] do_raw_spin_lock+0x177/0x190 [11700.166355] _raw_spin_lock+0x21/0x30 [11700.167825] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11700.170667] ? do_raw_spin_unlock+0x75/0x190 [11700.172359] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11700.175200] ? do_wait_intr+0xf0/0xf0 [11700.176768] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11700.179493] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11700.182318] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11700.184934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11700.187235] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11700.189706] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11700.191923] mdc_intent_lock+0x353/0x7e0 [mdc] [11700.193701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11700.196498] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11700.199086] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11700.200535] lmv_intent_open+0x338/0x1150 [lmv] [11700.202759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11700.205475] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11700.208074] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11700.210333] ll_intent_lock+0xc5/0xaa0 [lustre] [11700.212623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11700.215279] ll_lookup_it+0xe0e/0x2270 [lustre] [11700.217364] ? do_raw_spin_unlock+0x75/0x190 [11700.219305] ? _raw_spin_unlock+0x12/0x30 [11700.220710] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11700.223241] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11700.225515] ll_atomic_open+0x36a/0xd70 [lustre] [11700.227752] lookup_open+0x356/0xb40 [11700.229544] do_last+0x313/0xfc0 [11700.230962] ? nd_jump_root+0xe5/0x160 [11700.232724] ? path_init+0x437/0x520 [11700.234601] path_openat+0xf7/0x500 [11700.236362] ? do_raw_spin_unlock+0x75/0x190 [11700.238234] do_filp_open+0x99/0x140 [11700.239938] ? getname_flags+0x6e/0x330 [11700.242313] ? __check_object_size+0xff/0x256 [11700.244350] ? do_raw_spin_unlock+0x75/0x190 [11700.246347] ? _raw_spin_unlock+0x12/0x30 [11700.248577] do_sys_openat2+0x2b4/0x410 [11700.250114] do_sys_open+0x73/0xa0 [11700.251701] __x64_sys_openat+0x24/0x30 [11700.253757] do_syscall_64+0xc1/0x440 [11700.255652] entry_SYSCALL_64_after_hwframe+0x49/0xae [11700.257949] RIP: 0033:0x7f1a63983332 [11700.259622] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11700.268613] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11700.272551] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11700.275804] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11700.279390] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11700.282774] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11700.286614] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11704.058119] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [11704.064104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11704.105347] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11704.114256] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11704.127189] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11704.131272] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11704.147239] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11704.153687] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11704.160537] RDX: 0000000000007cc2 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11704.164633] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11704.167359] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11704.171543] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11704.177569] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11704.187461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11704.190263] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11704.193385] Call Trace: [11704.194521] [11704.195335] ? show_regs.cold.9+0x22/0x2f [11704.197275] ? watchdog_timer_fn.cold.6+0x62/0x10a [11704.199373] ? __hrtimer_run_queues+0x1c2/0x4d0 [11704.201651] ? watchdog+0x40/0x40 [11704.203133] ? hrtimer_interrupt+0x128/0x2c0 [11704.205001] ? smp_apic_timer_interrupt+0xa6/0x270 [11704.207232] ? apic_timer_interrupt+0xf/0x20 [11704.211064] [11704.215040] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11704.217046] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11704.219786] do_raw_spin_lock+0x177/0x190 [11704.227592] _raw_spin_lock+0x21/0x30 [11704.229337] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11704.234757] obd_mod_free+0x46/0x140 [ptlrpc] [11704.239326] mdc_commit_open+0x91/0xb0 [mdc] [11704.247288] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11704.249974] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11704.252745] after_reply+0xe2c/0x1780 [ptlrpc] [11704.254800] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11704.257418] ? __next_timer_interrupt+0x160/0x160 [11704.260612] ptlrpcd+0xa82/0xe40 [ptlrpc] [11704.266081] ? do_wait_intr+0xf0/0xf0 [11704.268170] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11704.273162] kthread+0x1d1/0x200 [11704.279697] ? set_kthread_struct+0x70/0x70 [11704.281207] ret_from_fork+0x1f/0x30 [11704.804171] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9681s! [11704.816048] Showing busy workqueues and worker pools: [11704.819737] workqueue events: flags=0x0 [11704.821719] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11704.825873] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11704.830618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11704.834859] pending: sec_gc_main [ptlrpc] [11704.837318] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11704.840778] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11704.845323] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11704.848659] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11704.857832] workqueue events_long: flags=0x0 [11704.860520] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11704.863742] pending: lu_objects_flush [obdclass] [11704.867662] workqueue events_unbound: flags=0x2 [11704.870660] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11704.873810] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11704.878655] workqueue events_power_efficient: flags=0x80 [11704.881192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11704.890225] pending: neigh_periodic_work, do_cache_clean [11704.893327] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11704.897145] pending: neigh_periodic_work, check_lifetime [11704.899972] workqueue mm_percpu_wq: flags=0x8 [11704.902470] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11704.905710] pending: vmstat_update [11704.907669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11704.911061] pending: vmstat_update [11704.913398] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11704.917519] pending: vmstat_update [11704.919936] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11704.923546] pending: vmstat_update [11704.925572] workqueue writeback: flags=0x4a [11704.927560] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11704.930383] pending: wb_workfn, wb_workfn [11704.932651] workqueue xprtiod: flags=0xa [11704.934538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11704.944636] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11704.955235] workqueue ipv6_addrconf: flags=0x40008 [11704.957156] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11704.961706] pending: addrconf_verify_work [11704.964566] workqueue knbd0-recv: flags=0x1a [11704.966749] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11704.972716] in-flight: 647:recv_work [11704.975803] workqueue ptlrpc_pinger: flags=0x2 [11704.980085] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11704.982379] pending: ptlrpc_pinger_main [ptlrpc] [11704.993970] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11695s workers=2 idle: 652 [11720.057078] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [11720.057046] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11720.057108] Modules linked in: [11720.059793] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11720.059986] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11720.059994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11720.059997] RIP: 0010:native_safe_halt+0x2e/0x50 [11720.060030] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11720.060034] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11720.060039] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11720.060041] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11720.060044] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11720.060046] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11720.060048] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11720.060051] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11720.060054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11720.060056] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11720.060063] Call Trace: [11720.060074] [11720.060080] ? show_regs.cold.9+0x22/0x2f [11720.060088] ? watchdog_timer_fn.cold.6+0x62/0x10a [11720.060094] ? __hrtimer_run_queues+0x1c2/0x4d0 [11720.060098] ? watchdog+0x40/0x40 [11720.060102] ? hrtimer_interrupt+0x128/0x2c0 [11720.060105] ? smp_apic_timer_interrupt+0xa6/0x270 [11720.060112] ? apic_timer_interrupt+0xf/0x20 [11720.060116] [11720.060118] ? native_safe_halt+0x2e/0x50 [11720.060121] kvm_wait+0xcd/0xe0 [11720.060127] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11720.060134] do_raw_spin_lock+0x177/0x190 [11720.060137] _raw_spin_lock+0x21/0x30 [11720.060142] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11720.062391] ? do_raw_spin_unlock+0x75/0x190 [11720.062401] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11720.062598] ? do_wait_intr+0xf0/0xf0 [11720.062606] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11720.062794] mdc_close+0x53b/0xd70 [mdc] [11720.062835] lmv_close+0x18d/0x490 [lmv] [11720.062863] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11720.063054] ll_md_real_close+0xa5/0x2a0 [lustre] [11720.063131] ll_md_close+0x274/0x800 [lustre] [11720.063210] ll_file_release+0x128/0xc50 [lustre] [11720.080106] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11720.080184] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11720.080192] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11720.080195] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11720.080209] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11720.080213] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11720.080217] RAX: 000000000000430c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11720.080220] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11720.080222] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11720.080224] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11720.080227] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11720.080230] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11720.080233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11720.080235] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11720.080255] Call Trace: [11720.080261] [11720.080267] ? show_regs.cold.9+0x22/0x2f [11720.080276] ? watchdog_timer_fn.cold.6+0x62/0x10a [11720.080283] ? __hrtimer_run_queues+0x1c2/0x4d0 [11720.080287] ? watchdog+0x40/0x40 [11720.080291] ? hrtimer_interrupt+0x128/0x2c0 [11720.080294] ? smp_apic_timer_interrupt+0xa6/0x270 [11720.080303] ? apic_timer_interrupt+0xf/0x20 [11720.080307] [11720.080309] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11720.080315] do_raw_spin_lock+0x177/0x190 [11720.080320] _raw_spin_lock+0x21/0x30 [11720.080324] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11720.081679] ? do_raw_spin_unlock+0x75/0x190 [11720.081836] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11720.082180] ? do_wait_intr+0xf0/0xf0 [11720.082189] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11720.096730] __fput+0x10d/0x440 [11720.096753] ____fput+0x12/0x20 [11720.096758] task_work_run+0xc6/0x110 [11720.096767] exit_to_usermode_loop+0x1dd/0x1f0 [11720.096775] do_syscall_64+0x3d6/0x440 [11720.096780] entry_SYSCALL_64_after_hwframe+0x49/0xae [11720.096789] RIP: 0033:0x7f1a63983cd8 [11720.096839] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11720.096844] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11720.096850] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11720.096852] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11720.096854] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11720.096856] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11720.096858] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11720.639613] mdc_close+0x53b/0xd70 [mdc] [11720.646070] lmv_close+0x18d/0x490 [lmv] [11720.649078] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11720.651837] ll_md_real_close+0xa5/0x2a0 [lustre] [11720.654751] ll_md_close+0x274/0x800 [lustre] [11720.656803] ll_file_release+0x128/0xc50 [lustre] [11720.659040] __fput+0x10d/0x440 [11720.661709] ____fput+0x12/0x20 [11720.663958] task_work_run+0xc6/0x110 [11720.666070] exit_to_usermode_loop+0x1dd/0x1f0 [11720.668359] do_syscall_64+0x3d6/0x440 [11720.673894] entry_SYSCALL_64_after_hwframe+0x49/0xae [11720.678862] RIP: 0033:0x7f1a63983cd8 [11720.682898] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11720.698801] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11720.703800] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11720.708871] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11720.717398] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11720.725401] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11720.739405] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11732.055680] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [11732.058040] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [11732.068754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11732.068840] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11732.068848] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11732.068850] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11732.068871] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11732.068875] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11732.068880] RAX: 0000000000001b7f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11732.068882] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11732.068885] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11732.068887] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11732.068890] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11732.068893] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11732.068897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11732.068899] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11732.068908] Call Trace: [11732.068913] [11732.068919] ? show_regs.cold.9+0x22/0x2f [11732.068929] ? watchdog_timer_fn.cold.6+0x62/0x10a [11732.068936] ? __hrtimer_run_queues+0x1c2/0x4d0 [11732.068940] ? watchdog+0x40/0x40 [11732.068945] ? hrtimer_interrupt+0x128/0x2c0 [11732.068949] ? smp_apic_timer_interrupt+0xa6/0x270 [11732.068958] ? apic_timer_interrupt+0xf/0x20 [11732.068962] [11732.068965] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11732.068971] do_raw_spin_lock+0x177/0x190 [11732.068976] _raw_spin_lock+0x21/0x30 [11732.068981] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11732.069474] ? do_raw_spin_unlock+0x75/0x190 [11732.069481] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11732.069676] ? do_wait_intr+0xf0/0xf0 [11732.069684] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11732.069889] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11732.074048] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11732.074216] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11732.074224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11732.074228] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11732.074244] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11732.074248] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11732.074253] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11732.074256] RDX: 000000000000185e RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11732.074258] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11732.074261] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11732.074263] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11732.074267] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11732.074270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11732.074275] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11732.074282] Call Trace: [11732.074287] [11732.074292] ? show_regs.cold.9+0x22/0x2f [11732.074302] ? watchdog_timer_fn.cold.6+0x62/0x10a [11732.074313] ? __hrtimer_run_queues+0x1c2/0x4d0 [11732.074316] ? watchdog+0x40/0x40 [11732.074320] ? hrtimer_interrupt+0x128/0x2c0 [11732.074324] ? smp_apic_timer_interrupt+0xa6/0x270 [11732.075056] ? apic_timer_interrupt+0xf/0x20 [11732.075064] [11732.075067] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11732.075073] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11732.075078] do_raw_spin_lock+0x177/0x190 [11732.075083] _raw_spin_lock+0x21/0x30 [11732.075088] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11732.076061] obd_mod_free+0x46/0x140 [ptlrpc] [11732.076610] mdc_commit_open+0x91/0xb0 [mdc] [11732.076657] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11732.077137] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11732.077878] after_reply+0xe2c/0x1780 [ptlrpc] [11732.078189] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11732.078411] ? __next_timer_interrupt+0x160/0x160 [11732.078421] ptlrpcd+0xa82/0xe40 [ptlrpc] [11732.078640] ? do_wait_intr+0xf0/0xf0 [11732.078653] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11732.078914] kthread+0x1d1/0x200 [11732.078919] ? set_kthread_struct+0x70/0x70 [11732.078923] ret_from_fork+0x1f/0x30 [11732.699887] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11732.702194] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11732.709531] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11732.715212] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11732.718752] mdc_intent_lock+0x353/0x7e0 [mdc] [11732.722777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11732.730639] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11732.738900] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11732.744456] lmv_intent_open+0x338/0x1150 [lmv] [11732.746969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11732.757637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11732.764866] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11732.767309] ll_intent_lock+0xc5/0xaa0 [lustre] [11732.769827] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11732.772717] ll_lookup_it+0xe0e/0x2270 [lustre] [11732.776934] ? do_raw_spin_unlock+0x75/0x190 [11732.778185] ? _raw_spin_unlock+0x12/0x30 [11732.779473] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11732.786035] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11732.790793] ll_atomic_open+0x36a/0xd70 [lustre] [11732.792938] lookup_open+0x356/0xb40 [11732.795511] do_last+0x313/0xfc0 [11732.797178] ? nd_jump_root+0xe5/0x160 [11732.803823] ? path_init+0x437/0x520 [11732.806611] path_openat+0xf7/0x500 [11732.808664] ? do_raw_spin_unlock+0x75/0x190 [11732.810387] do_filp_open+0x99/0x140 [11732.813904] ? getname_flags+0x6e/0x330 [11732.817019] ? __check_object_size+0xff/0x256 [11732.820249] ? do_raw_spin_unlock+0x75/0x190 [11732.821693] ? _raw_spin_unlock+0x12/0x30 [11732.822947] do_sys_openat2+0x2b4/0x410 [11732.824960] do_sys_open+0x73/0xa0 [11732.825840] __x64_sys_openat+0x24/0x30 [11732.827747] do_syscall_64+0xc1/0x440 [11732.829339] entry_SYSCALL_64_after_hwframe+0x49/0xae [11732.831656] RIP: 0033:0x7f1a63983332 [11732.833487] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11732.851355] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11732.858661] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11732.867619] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11732.870715] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11732.875437] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11732.879779] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11735.009885] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9711s! [11735.016662] Showing busy workqueues and worker pools: [11735.019885] workqueue events: flags=0x0 [11735.021749] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11735.024681] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11735.028495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11735.031658] pending: sec_gc_main [ptlrpc] [11735.034776] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11735.037428] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11735.049374] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11735.053629] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11735.062488] workqueue events_long: flags=0x0 [11735.070808] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11735.074872] pending: lu_objects_flush [obdclass] [11735.077191] workqueue events_unbound: flags=0x2 [11735.079544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11735.082648] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11735.090622] workqueue events_power_efficient: flags=0x80 [11735.094764] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11735.098910] pending: neigh_periodic_work, do_cache_clean [11735.105564] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11735.108947] pending: neigh_periodic_work, check_lifetime [11735.116353] workqueue mm_percpu_wq: flags=0x8 [11735.118874] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11735.124838] pending: vmstat_update [11735.127666] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11735.133195] pending: vmstat_update [11735.134693] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11735.141553] pending: vmstat_update [11735.144016] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11735.148605] pending: vmstat_update [11735.153650] workqueue writeback: flags=0x4a [11735.155324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11735.164952] pending: wb_workfn, wb_workfn [11735.168583] workqueue xprtiod: flags=0xa [11735.170931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11735.174854] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11735.186927] workqueue ipv6_addrconf: flags=0x40008 [11735.191683] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11735.193950] pending: addrconf_verify_work [11735.198706] workqueue knbd0-recv: flags=0x1a [11735.200970] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11735.204006] in-flight: 647:recv_work [11735.205960] workqueue ptlrpc_pinger: flags=0x2 [11735.207662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11735.210482] pending: ptlrpc_pinger_main [ptlrpc] [11735.213059] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11725s workers=2 idle: 652 [11748.057499] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [11748.060172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11748.066290] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11748.095537] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11748.095552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11748.095556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.095574] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11748.095577] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11748.096147] RAX: 00000000000007c8 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11748.096150] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11748.096152] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11748.096153] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11748.096155] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11748.096157] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11748.096159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11748.096161] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11748.096167] Call Trace: [11748.096172] [11748.096176] ? show_regs.cold.9+0x22/0x2f [11748.096184] ? watchdog_timer_fn.cold.6+0x62/0x10a [11748.096189] ? __hrtimer_run_queues+0x1c2/0x4d0 [11748.096192] ? watchdog+0x40/0x40 [11748.096195] ? hrtimer_interrupt+0x128/0x2c0 [11748.096197] ? smp_apic_timer_interrupt+0xa6/0x270 [11748.096204] ? apic_timer_interrupt+0xf/0x20 [11748.096207] [11748.096209] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.096214] do_raw_spin_lock+0x177/0x190 [11748.096218] _raw_spin_lock+0x21/0x30 [11748.096221] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11748.096498] ? do_raw_spin_unlock+0x75/0x190 [11748.096504] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11748.097813] ? do_wait_intr+0xf0/0xf0 [11748.097820] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11748.097949] mdc_close+0x53b/0xd70 [mdc] [11748.098001] lmv_close+0x18d/0x490 [lmv] [11748.098036] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11748.108068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11748.108154] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11748.108161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11748.108165] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.108180] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11748.108184] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11748.108189] RAX: 0000000000007792 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11748.108191] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11748.108194] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11748.108197] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11748.108199] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11748.108202] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11748.108205] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11748.108207] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11748.108216] Call Trace: [11748.108220] [11748.108226] ? show_regs.cold.9+0x22/0x2f [11748.108235] ? watchdog_timer_fn.cold.6+0x62/0x10a [11748.108241] ? __hrtimer_run_queues+0x1c2/0x4d0 [11748.108246] ? watchdog+0x40/0x40 [11748.108249] ? hrtimer_interrupt+0x128/0x2c0 [11748.108253] ? smp_apic_timer_interrupt+0xa6/0x270 [11748.108263] ? apic_timer_interrupt+0xf/0x20 [11748.108267] [11748.108270] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.108276] do_raw_spin_lock+0x177/0x190 [11748.108281] _raw_spin_lock+0x21/0x30 [11748.108286] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11748.109036] ? do_raw_spin_unlock+0x75/0x190 [11748.114109] ll_md_real_close+0xa5/0x2a0 [lustre] [11748.114249] ll_md_close+0x274/0x800 [lustre] [11748.114338] ll_file_release+0x128/0xc50 [lustre] [11748.114427] __fput+0x10d/0x440 [11748.114438] ____fput+0x12/0x20 [11748.114442] task_work_run+0xc6/0x110 [11748.114450] exit_to_usermode_loop+0x1dd/0x1f0 [11748.114457] do_syscall_64+0x3d6/0x440 [11748.114461] entry_SYSCALL_64_after_hwframe+0x49/0xae [11748.114470] RIP: 0033:0x7f1a63983cd8 [11748.114478] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11748.114481] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11748.114487] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11748.114489] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11748.114492] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11748.114494] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11748.114496] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11748.441640] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11748.444871] ? do_wait_intr+0xf0/0xf0 [11748.447293] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11748.454360] mdc_close+0x53b/0xd70 [mdc] [11748.459110] lmv_close+0x18d/0x490 [lmv] [11748.461802] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11748.465794] ll_md_real_close+0xa5/0x2a0 [lustre] [11748.468426] ll_md_close+0x274/0x800 [lustre] [11748.471602] ll_file_release+0x128/0xc50 [lustre] [11748.476714] __fput+0x10d/0x440 [11748.478655] ____fput+0x12/0x20 [11748.480525] task_work_run+0xc6/0x110 [11748.482857] exit_to_usermode_loop+0x1dd/0x1f0 [11748.485331] do_syscall_64+0x3d6/0x440 [11748.488330] entry_SYSCALL_64_after_hwframe+0x49/0xae [11748.491329] RIP: 0033:0x7f1a63983cd8 [11748.493369] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11748.508536] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11748.513741] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11748.517258] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11748.523729] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11748.527532] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11748.531871] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11760.053020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11760.057438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11760.058057] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11760.086042] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11760.092637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11760.092717] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11760.092724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11760.092727] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11760.093145] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11760.093150] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11760.093154] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11760.093156] RDX: 0000000000007984 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11760.093159] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11760.093161] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11760.093163] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11760.093167] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11760.093170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11760.093172] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11760.093181] Call Trace: [11760.093186] [11760.093192] ? show_regs.cold.9+0x22/0x2f [11760.093201] ? watchdog_timer_fn.cold.6+0x62/0x10a [11760.093207] ? __hrtimer_run_queues+0x1c2/0x4d0 [11760.093211] ? watchdog+0x40/0x40 [11760.093215] ? hrtimer_interrupt+0x128/0x2c0 [11760.093219] ? smp_apic_timer_interrupt+0xa6/0x270 [11760.093229] ? apic_timer_interrupt+0xf/0x20 [11760.093234] [11760.093237] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11760.093241] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11760.093245] do_raw_spin_lock+0x177/0x190 [11760.093250] _raw_spin_lock+0x21/0x30 [11760.093255] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11760.093588] obd_mod_free+0x46/0x140 [ptlrpc] [11760.093915] mdc_commit_open+0x91/0xb0 [mdc] [11760.093974] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11760.094188] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11760.094381] after_reply+0xe2c/0x1780 [ptlrpc] [11760.094586] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11760.100971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11760.100987] RIP: 0010:native_safe_halt+0x2e/0x50 [11760.101006] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11760.101012] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11760.101019] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11760.101022] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11760.101024] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11760.101027] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11760.101030] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11760.101033] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11760.101037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11760.101039] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11760.101049] Call Trace: [11760.101056] [11760.101063] ? show_regs.cold.9+0x22/0x2f [11760.101076] ? watchdog_timer_fn.cold.6+0x62/0x10a [11760.101085] ? __hrtimer_run_queues+0x1c2/0x4d0 [11760.101090] ? watchdog+0x40/0x40 [11760.101095] ? hrtimer_interrupt+0x128/0x2c0 [11760.101099] ? smp_apic_timer_interrupt+0xa6/0x270 [11760.101105] ? apic_timer_interrupt+0xf/0x20 [11760.101110] [11760.101113] ? native_safe_halt+0x2e/0x50 [11760.101117] kvm_wait+0xcd/0xe0 [11760.101125] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11760.101135] do_raw_spin_lock+0x177/0x190 [11760.101140] _raw_spin_lock+0x21/0x30 [11760.101145] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11760.101428] ? do_raw_spin_unlock+0x75/0x190 [11760.101433] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11760.101624] ? do_wait_intr+0xf0/0xf0 [11760.101633] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11760.101832] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11760.102591] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11760.102804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11760.104055] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11760.104098] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11760.104131] mdc_intent_lock+0x353/0x7e0 [mdc] [11760.104160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11760.104245] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11760.129618] ? __next_timer_interrupt+0x160/0x160 [11760.129643] ptlrpcd+0xa82/0xe40 [ptlrpc] [11760.130243] ? do_wait_intr+0xf0/0xf0 [11760.130255] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11760.130551] kthread+0x1d1/0x200 [11760.130562] ? set_kthread_struct+0x70/0x70 [11760.130567] ret_from_fork+0x1f/0x30 [11760.523430] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11760.525648] lmv_intent_open+0x338/0x1150 [lmv] [11760.527573] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11760.529616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11760.532659] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11760.534971] ll_intent_lock+0xc5/0xaa0 [lustre] [11760.537121] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11760.540052] ll_lookup_it+0xe0e/0x2270 [lustre] [11760.542248] ? do_raw_spin_unlock+0x75/0x190 [11760.545817] ? _raw_spin_unlock+0x12/0x30 [11760.547716] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11760.552215] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11760.554441] ll_atomic_open+0x36a/0xd70 [lustre] [11760.557283] lookup_open+0x356/0xb40 [11760.559348] do_last+0x313/0xfc0 [11760.561705] ? nd_jump_root+0xe5/0x160 [11760.563176] ? path_init+0x437/0x520 [11760.564723] path_openat+0xf7/0x500 [11760.566079] ? do_raw_spin_unlock+0x75/0x190 [11760.569474] do_filp_open+0x99/0x140 [11760.572101] ? getname_flags+0x6e/0x330 [11760.577317] ? __check_object_size+0xff/0x256 [11760.579539] ? do_raw_spin_unlock+0x75/0x190 [11760.581456] ? _raw_spin_unlock+0x12/0x30 [11760.583488] do_sys_openat2+0x2b4/0x410 [11760.586940] do_sys_open+0x73/0xa0 [11760.590152] __x64_sys_openat+0x24/0x30 [11760.592191] do_syscall_64+0xc1/0x440 [11760.593716] entry_SYSCALL_64_after_hwframe+0x49/0xae [11760.595932] RIP: 0033:0x7f1a63983332 [11760.597717] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11760.604946] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11760.608509] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11760.611516] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11760.614620] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11760.617461] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11760.620248] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11765.729104] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9742s! [11765.734194] Showing busy workqueues and worker pools: [11765.737659] workqueue events: flags=0x0 [11765.740439] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11765.744561] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11765.749609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11765.753407] pending: sec_gc_main [ptlrpc] [11765.756659] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11765.760986] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11765.767931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11765.772219] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11765.777828] workqueue events_long: flags=0x0 [11765.780098] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11765.783561] pending: lu_objects_flush [obdclass] [11765.786171] workqueue events_unbound: flags=0x2 [11765.788277] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11765.791176] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11765.796256] workqueue events_power_efficient: flags=0x80 [11765.798615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11765.801619] pending: neigh_periodic_work, do_cache_clean [11765.804500] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11765.807439] pending: neigh_periodic_work, check_lifetime [11765.810881] workqueue mm_percpu_wq: flags=0x8 [11765.813131] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11765.816229] pending: vmstat_update [11765.817936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11765.820556] pending: vmstat_update [11765.822120] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11765.824484] pending: vmstat_update [11765.826173] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11765.828570] pending: vmstat_update [11765.830361] workqueue writeback: flags=0x4a [11765.832071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11765.834551] pending: wb_workfn, wb_workfn [11765.836391] workqueue xprtiod: flags=0xa [11765.837909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11765.840372] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11765.843371] workqueue ipv6_addrconf: flags=0x40008 [11765.845285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11765.848415] pending: addrconf_verify_work [11765.850422] workqueue knbd0-recv: flags=0x1a [11765.852566] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11765.855898] in-flight: 647:recv_work [11765.858132] workqueue ptlrpc_pinger: flags=0x2 [11765.860459] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11765.863535] pending: ptlrpc_pinger_main [ptlrpc] [11765.866644] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11755s workers=2 idle: 652 [11776.056064] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [11776.057033] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11776.058820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11776.058898] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11776.058906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11776.058909] RIP: 0010:native_safe_halt+0x2e/0x50 [11776.058924] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11776.058929] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11776.058934] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11776.058936] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11776.058938] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11776.058940] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11776.058943] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11776.058946] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11776.058949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11776.058952] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11776.058960] Call Trace: [11776.058965] [11776.058972] ? show_regs.cold.9+0x22/0x2f [11776.058983] ? watchdog_timer_fn.cold.6+0x62/0x10a [11776.058989] ? __hrtimer_run_queues+0x1c2/0x4d0 [11776.058994] ? watchdog+0x40/0x40 [11776.058998] ? hrtimer_interrupt+0x128/0x2c0 [11776.059002] ? smp_apic_timer_interrupt+0xa6/0x270 [11776.059009] ? apic_timer_interrupt+0xf/0x20 [11776.059014] [11776.059017] ? native_safe_halt+0x2e/0x50 [11776.059021] kvm_wait+0xcd/0xe0 [11776.059029] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11776.059038] do_raw_spin_lock+0x177/0x190 [11776.059042] _raw_spin_lock+0x21/0x30 [11776.059046] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11776.059324] ? do_raw_spin_unlock+0x75/0x190 [11776.059330] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11776.059529] ? do_wait_intr+0xf0/0xf0 [11776.059537] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11776.059749] mdc_close+0x53b/0xd70 [mdc] [11776.059802] lmv_close+0x18d/0x490 [lmv] [11776.059836] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11776.059955] ll_md_real_close+0xa5/0x2a0 [lustre] [11776.060039] ll_md_close+0x274/0x800 [lustre] [11776.060137] ll_file_release+0x128/0xc50 [lustre] [11776.060221] __fput+0x10d/0x440 [11776.060231] ____fput+0x12/0x20 [11776.060236] task_work_run+0xc6/0x110 [11776.060245] exit_to_usermode_loop+0x1dd/0x1f0 [11776.060254] do_syscall_64+0x3d6/0x440 [11776.060260] entry_SYSCALL_64_after_hwframe+0x49/0xae [11776.060267] RIP: 0033:0x7f1a63983cd8 [11776.060273] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11776.060278] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11776.060282] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11776.060284] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11776.060287] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11776.060289] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11776.060291] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11776.444165] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11776.489139] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11776.500206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11776.513676] RIP: 0010:native_safe_halt+0x2e/0x50 [11776.517073] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11776.533286] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11776.538285] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11776.543252] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [11776.548173] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11776.551514] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11776.554652] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11776.557895] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11776.561481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11776.564813] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11776.568255] Call Trace: [11776.569499] [11776.570439] ? show_regs.cold.9+0x22/0x2f [11776.572249] ? watchdog_timer_fn.cold.6+0x62/0x10a [11776.574332] ? __hrtimer_run_queues+0x1c2/0x4d0 [11776.576262] ? watchdog+0x40/0x40 [11776.577641] ? hrtimer_interrupt+0x128/0x2c0 [11776.585494] ? smp_apic_timer_interrupt+0xa6/0x270 [11776.593307] ? apic_timer_interrupt+0xf/0x20 [11776.600377] [11776.604650] ? native_safe_halt+0x2e/0x50 [11776.610671] kvm_wait+0xcd/0xe0 [11776.613239] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11776.623155] do_raw_spin_lock+0x177/0x190 [11776.629039] _raw_spin_lock+0x21/0x30 [11776.638010] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11776.641293] ? do_raw_spin_unlock+0x75/0x190 [11776.645343] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11776.649560] ? do_wait_intr+0xf0/0xf0 [11776.659105] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11776.667402] mdc_close+0x53b/0xd70 [mdc] [11776.670601] lmv_close+0x18d/0x490 [lmv] [11776.676833] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11776.679656] ll_md_real_close+0xa5/0x2a0 [lustre] [11776.683447] ll_md_close+0x274/0x800 [lustre] [11776.692480] ll_file_release+0x128/0xc50 [lustre] [11776.698457] __fput+0x10d/0x440 [11776.705502] ____fput+0x12/0x20 [11776.708597] task_work_run+0xc6/0x110 [11776.715157] exit_to_usermode_loop+0x1dd/0x1f0 [11776.717526] do_syscall_64+0x3d6/0x440 [11776.720045] entry_SYSCALL_64_after_hwframe+0x49/0xae [11776.722960] RIP: 0033:0x7f1a63983cd8 [11776.724998] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11776.735421] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11776.746514] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11776.755152] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11776.766937] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11776.775590] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11776.787503] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11788.063145] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11788.073515] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11788.100939] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11788.122562] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11788.134061] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11788.140210] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11788.168927] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11788.177567] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11788.188731] RDX: 0000000000006005 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11788.198694] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11788.208078] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11788.213192] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11788.219995] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11788.226677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11788.233877] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11788.238461] Call Trace: [11788.239526] [11788.240441] ? show_regs.cold.9+0x22/0x2f [11788.245041] ? watchdog_timer_fn.cold.6+0x62/0x10a [11788.248427] ? __hrtimer_run_queues+0x1c2/0x4d0 [11788.250572] ? watchdog+0x40/0x40 [11788.260469] ? hrtimer_interrupt+0x128/0x2c0 [11788.266114] ? smp_apic_timer_interrupt+0xa6/0x270 [11788.270509] ? apic_timer_interrupt+0xf/0x20 [11788.278618] [11788.279453] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11788.282418] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11788.291118] do_raw_spin_lock+0x177/0x190 [11788.298393] _raw_spin_lock+0x21/0x30 [11788.300762] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11788.303238] obd_mod_free+0x46/0x140 [ptlrpc] [11788.306001] mdc_commit_open+0x91/0xb0 [mdc] [11788.306993] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11788.309955] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11788.311763] after_reply+0xe2c/0x1780 [ptlrpc] [11788.314628] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11788.316843] ? __next_timer_interrupt+0x160/0x160 [11788.318629] ptlrpcd+0xa82/0xe40 [ptlrpc] [11788.320420] ? do_wait_intr+0xf0/0xf0 [11788.321641] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11788.324835] kthread+0x1d1/0x200 [11788.325949] ? set_kthread_struct+0x70/0x70 [11788.327489] ret_from_fork+0x1f/0x30 [11792.060147] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [11792.064646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11792.083597] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11792.087837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11792.091736] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11792.094362] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11792.108103] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11792.115256] RAX: 0000000000007d85 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11792.121356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11792.128029] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11792.131424] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11792.137021] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11792.142952] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11792.147359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11792.149828] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11792.153708] Call Trace: [11792.157341] [11792.158185] ? show_regs.cold.9+0x22/0x2f [11792.164282] ? watchdog_timer_fn.cold.6+0x62/0x10a [11792.166464] ? __hrtimer_run_queues+0x1c2/0x4d0 [11792.168961] ? watchdog+0x40/0x40 [11792.170758] ? hrtimer_interrupt+0x128/0x2c0 [11792.176187] ? smp_apic_timer_interrupt+0xa6/0x270 [11792.178262] ? apic_timer_interrupt+0xf/0x20 [11792.183008] [11792.183856] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11792.190277] do_raw_spin_lock+0x177/0x190 [11792.192694] _raw_spin_lock+0x21/0x30 [11792.194734] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11792.204358] ? do_raw_spin_unlock+0x75/0x190 [11792.207202] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11792.211732] ? do_wait_intr+0xf0/0xf0 [11792.215234] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11792.217675] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11792.220005] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11792.223350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11792.229807] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11792.234198] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11792.236994] mdc_intent_lock+0x353/0x7e0 [mdc] [11792.239303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11792.248238] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11792.250819] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11792.257477] lmv_intent_open+0x338/0x1150 [lmv] [11792.262622] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11792.266302] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11792.275210] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11792.277919] ll_intent_lock+0xc5/0xaa0 [lustre] [11792.283379] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11792.285474] ll_lookup_it+0xe0e/0x2270 [lustre] [11792.287447] ? do_raw_spin_unlock+0x75/0x190 [11792.289212] ? _raw_spin_unlock+0x12/0x30 [11792.290685] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11792.292758] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11792.294419] ll_atomic_open+0x36a/0xd70 [lustre] [11792.296508] lookup_open+0x356/0xb40 [11792.297892] do_last+0x313/0xfc0 [11792.298999] ? nd_jump_root+0xe5/0x160 [11792.300195] ? path_init+0x437/0x520 [11792.301501] path_openat+0xf7/0x500 [11792.305251] ? do_raw_spin_unlock+0x75/0x190 [11792.306839] do_filp_open+0x99/0x140 [11792.308355] ? getname_flags+0x6e/0x330 [11792.309688] ? __check_object_size+0xff/0x256 [11792.312121] ? do_raw_spin_unlock+0x75/0x190 [11792.313803] ? _raw_spin_unlock+0x12/0x30 [11792.316465] do_sys_openat2+0x2b4/0x410 [11792.317655] do_sys_open+0x73/0xa0 [11792.318674] __x64_sys_openat+0x24/0x30 [11792.320423] do_syscall_64+0xc1/0x440 [11792.321644] entry_SYSCALL_64_after_hwframe+0x49/0xae [11792.328336] RIP: 0033:0x7f1a63983332 [11792.330014] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11792.353130] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11792.365403] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11792.370490] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11792.376406] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11792.382510] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11792.386958] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11795.936911] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9772s! [11795.940368] Showing busy workqueues and worker pools: [11795.946647] workqueue events: flags=0x0 [11795.948796] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11795.955448] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11795.959684] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11795.963127] pending: sec_gc_main [ptlrpc] [11795.968574] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11795.977120] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11795.990652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11795.994333] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11796.002308] workqueue events_long: flags=0x0 [11796.005045] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11796.009472] pending: lu_objects_flush [obdclass] [11796.012962] workqueue events_unbound: flags=0x2 [11796.019013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11796.022812] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11796.038105] workqueue events_power_efficient: flags=0x80 [11796.044665] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11796.052272] pending: neigh_periodic_work, do_cache_clean [11796.059316] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11796.064780] pending: neigh_periodic_work, check_lifetime [11796.070570] workqueue mm_percpu_wq: flags=0x8 [11796.073675] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11796.082071] pending: vmstat_update [11796.084393] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11796.092396] pending: vmstat_update [11796.094257] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11796.099261] pending: vmstat_update [11796.101651] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11796.105443] pending: vmstat_update [11796.107589] workqueue writeback: flags=0x4a [11796.109404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11796.112443] pending: wb_workfn, wb_workfn [11796.114284] workqueue xprtiod: flags=0xa [11796.116439] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11796.119302] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11796.123453] workqueue ipv6_addrconf: flags=0x40008 [11796.126274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11796.138186] pending: addrconf_verify_work [11796.143053] workqueue knbd0-recv: flags=0x1a [11796.146167] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11796.150247] in-flight: 647:recv_work [11796.152251] workqueue ptlrpc_pinger: flags=0x2 [11796.154684] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11796.160228] pending: ptlrpc_pinger_main [ptlrpc] [11796.166101] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11786s workers=2 idle: 652 [11802.665040] rcu: INFO: rcu_sched self-detected stall on CPU [11802.668255] rcu: 3-...!: (6152787 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [11802.675452] (t=9780268 jiffies g=269633 q=3465) [11802.678008] rcu: rcu_sched kthread starved for 9779917 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [11802.687158] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11802.691787] rcu: RCU grace-period kthread stack dump: [11802.694110] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11802.702622] Call Trace: [11802.703825] __schedule+0x351/0xcb0 [11802.706245] schedule+0xc0/0x180 [11802.707986] schedule_timeout+0xb4/0x190 [11802.712008] ? do_raw_spin_unlock+0x75/0x190 [11802.713885] ? __next_timer_interrupt+0x160/0x160 [11802.716092] ? prepare_to_swait_event+0xf3/0x140 [11802.718243] rcu_gp_kthread+0x8dd/0x15b0 [11802.720929] ? force_qs_rnp+0x2c0/0x2c0 [11802.724910] kthread+0x1d1/0x200 [11802.726477] ? set_kthread_struct+0x70/0x70 [11802.729023] ret_from_fork+0x1f/0x30 [11802.730687] rcu: Stack dump where RCU GP kthread last ran: [11802.736269] Sending NMI from CPU 3 to CPUs 2: [11802.738901] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [11802.739863] NMI backtrace for cpu 3 [11802.748597] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11802.753483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11802.757404] Call Trace: [11802.758655] [11802.759553] ? dump_stack+0xbb/0x10e [11802.761072] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11802.763150] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11802.765218] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11802.767161] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11802.769196] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11802.771252] ? rcu_dump_cpu_stacks+0x11b/0x171 [11802.773099] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11802.775759] ? update_process_times+0x6d/0xc0 [11802.777635] ? tick_sched_handle.isra.18+0x27/0x90 [11802.779771] ? tick_sched_timer+0x53/0xb0 [11802.782721] ? __hrtimer_run_queues+0x1c2/0x4d0 [11802.785626] ? tick_sched_handle.isra.18+0x90/0x90 [11802.787687] ? hrtimer_interrupt+0x128/0x2c0 [11802.790778] ? smp_apic_timer_interrupt+0xa6/0x270 [11802.793922] ? apic_timer_interrupt+0xf/0x20 [11802.796546] [11802.797400] ? native_safe_halt+0x2e/0x50 [11802.799394] ? kvm_wait+0xcd/0xe0 [11802.801647] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11802.804957] ? do_raw_spin_lock+0x177/0x190 [11802.807261] ? _raw_spin_lock+0x21/0x30 [11802.808356] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11802.810815] ? obd_mod_free+0x46/0x140 [ptlrpc] [11802.813759] ? mdc_commit_open+0x91/0xb0 [mdc] [11802.816618] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11802.819301] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11802.822042] ? after_reply+0xe2c/0x1780 [ptlrpc] [11802.824546] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11802.826404] ? __next_timer_interrupt+0x160/0x160 [11802.829355] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11802.831936] ? do_wait_intr+0xf0/0xf0 [11802.833731] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11802.836691] ? kthread+0x1d1/0x200 [11802.839206] ? set_kthread_struct+0x70/0x70 [11802.841432] ? ret_from_fork+0x1f/0x30 [11804.056574] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11804.068547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11804.106640] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11804.113488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11804.116644] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11804.127439] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11804.143405] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11804.149442] RAX: 000000000000490e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11804.159869] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11804.165460] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11804.169548] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11804.175629] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11804.180579] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11804.185619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11804.188600] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11804.198572] Call Trace: [11804.201835] [11804.204187] ? show_regs.cold.9+0x22/0x2f [11804.208179] ? watchdog_timer_fn.cold.6+0x62/0x10a [11804.214528] ? __hrtimer_run_queues+0x1c2/0x4d0 [11804.222160] ? watchdog+0x40/0x40 [11804.225185] ? hrtimer_interrupt+0x128/0x2c0 [11804.228220] ? smp_apic_timer_interrupt+0xa6/0x270 [11804.229952] ? apic_timer_interrupt+0xf/0x20 [11804.231382] [11804.237033] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11804.240291] do_raw_spin_lock+0x177/0x190 [11804.242624] _raw_spin_lock+0x21/0x30 [11804.244010] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11804.246066] ? do_raw_spin_unlock+0x75/0x190 [11804.247401] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11804.255741] ? do_wait_intr+0xf0/0xf0 [11804.256752] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11804.259359] mdc_close+0x53b/0xd70 [mdc] [11804.260489] lmv_close+0x18d/0x490 [lmv] [11804.261537] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11804.267714] ll_md_real_close+0xa5/0x2a0 [lustre] [11804.271051] ll_md_close+0x274/0x800 [lustre] [11804.275224] ll_file_release+0x128/0xc50 [lustre] [11804.279277] __fput+0x10d/0x440 [11804.280500] ____fput+0x12/0x20 [11804.281357] task_work_run+0xc6/0x110 [11804.282382] exit_to_usermode_loop+0x1dd/0x1f0 [11804.291953] do_syscall_64+0x3d6/0x440 [11804.294643] entry_SYSCALL_64_after_hwframe+0x49/0xae [11804.297276] RIP: 0033:0x7f1a63983cd8 [11804.309179] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11804.324555] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11804.328896] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11804.332903] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11804.337054] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11804.344175] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11804.350984] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11824.053034] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [11824.055705] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11824.080759] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11824.086453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11824.090369] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11824.093347] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11824.104788] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11824.108484] RAX: 00000000000013c2 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11824.111921] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11824.115509] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11824.118639] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11824.122367] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11824.125546] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11824.129266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11824.131591] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11824.134094] Call Trace: [11824.134874] [11824.135614] ? show_regs.cold.9+0x22/0x2f [11824.137320] ? watchdog_timer_fn.cold.6+0x62/0x10a [11824.139586] ? __hrtimer_run_queues+0x1c2/0x4d0 [11824.141752] ? watchdog+0x40/0x40 [11824.143911] ? hrtimer_interrupt+0x128/0x2c0 [11824.145944] ? smp_apic_timer_interrupt+0xa6/0x270 [11824.147836] ? apic_timer_interrupt+0xf/0x20 [11824.149729] [11824.150630] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11824.153890] do_raw_spin_lock+0x177/0x190 [11824.156852] _raw_spin_lock+0x21/0x30 [11824.158552] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11824.160893] ? do_raw_spin_unlock+0x75/0x190 [11824.162495] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11824.165606] ? do_wait_intr+0xf0/0xf0 [11824.167439] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11824.169801] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11824.173247] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11824.175589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11824.178895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11824.181875] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11824.183946] mdc_intent_lock+0x353/0x7e0 [mdc] [11824.186006] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11824.188553] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11824.191653] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11824.194148] lmv_intent_open+0x338/0x1150 [lmv] [11824.195975] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11824.198595] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11824.200758] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11824.202708] ll_intent_lock+0xc5/0xaa0 [lustre] [11824.204656] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11824.207150] ll_lookup_it+0xe0e/0x2270 [lustre] [11824.210096] ? do_raw_spin_unlock+0x75/0x190 [11824.212029] ? _raw_spin_unlock+0x12/0x30 [11824.213609] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11824.216091] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11824.218301] ll_atomic_open+0x36a/0xd70 [lustre] [11824.220516] lookup_open+0x356/0xb40 [11824.222092] do_last+0x313/0xfc0 [11824.223468] ? nd_jump_root+0xe5/0x160 [11824.225516] ? path_init+0x437/0x520 [11824.228691] path_openat+0xf7/0x500 [11824.230123] ? do_raw_spin_unlock+0x75/0x190 [11824.232207] do_filp_open+0x99/0x140 [11824.233755] ? getname_flags+0x6e/0x330 [11824.235430] ? __check_object_size+0xff/0x256 [11824.237440] ? do_raw_spin_unlock+0x75/0x190 [11824.239644] ? _raw_spin_unlock+0x12/0x30 [11824.241795] do_sys_openat2+0x2b4/0x410 [11824.243205] do_sys_open+0x73/0xa0 [11824.245390] __x64_sys_openat+0x24/0x30 [11824.246951] do_syscall_64+0xc1/0x440 [11824.248447] entry_SYSCALL_64_after_hwframe+0x49/0xae [11824.250715] RIP: 0033:0x7f1a63983332 [11824.252341] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11824.259796] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11824.264525] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11824.268682] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11824.272364] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11824.275807] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11824.279481] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11826.661403] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9803s! [11826.671293] Showing busy workqueues and worker pools: [11826.674512] workqueue events: flags=0x0 [11826.676626] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11826.680394] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11826.683744] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11826.686851] pending: sec_gc_main [ptlrpc] [11826.689268] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11826.692358] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11826.697302] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11826.701519] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11826.709071] workqueue events_long: flags=0x0 [11826.712675] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11826.719116] pending: lu_objects_flush [obdclass] [11826.724690] workqueue events_unbound: flags=0x2 [11826.727374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11826.730525] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11826.735684] workqueue events_power_efficient: flags=0x80 [11826.738523] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11826.741420] pending: neigh_periodic_work, do_cache_clean [11826.744109] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11826.748598] pending: neigh_periodic_work, check_lifetime [11826.752076] workqueue mm_percpu_wq: flags=0x8 [11826.754859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11826.759498] pending: vmstat_update [11826.761816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11826.765125] pending: vmstat_update [11826.767466] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11826.770928] pending: vmstat_update [11826.773098] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11826.777119] pending: vmstat_update [11826.779042] workqueue writeback: flags=0x4a [11826.781313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11826.785109] pending: wb_workfn, wb_workfn [11826.787156] workqueue xprtiod: flags=0xa [11826.788796] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11826.791967] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11826.796606] workqueue ipv6_addrconf: flags=0x40008 [11826.802083] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11826.807070] pending: addrconf_verify_work [11826.808978] workqueue knbd0-recv: flags=0x1a [11826.810300] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11826.813309] in-flight: 647:recv_work [11826.814840] workqueue ptlrpc_pinger: flags=0x2 [11826.818691] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11826.826036] pending: ptlrpc_pinger_main [ptlrpc] [11826.831808] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11816s workers=2 idle: 652 [11828.059024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11828.062908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11828.065667] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11828.086132] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11828.086149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11828.086153] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11828.086170] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11828.086175] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11828.086181] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11828.086183] RDX: 0000000000001e37 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11828.086185] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11828.086187] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11828.086190] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11828.086193] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11828.086208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11828.086211] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11828.086219] Call Trace: [11828.086224] [11828.086230] ? show_regs.cold.9+0x22/0x2f [11828.086247] ? watchdog_timer_fn.cold.6+0x62/0x10a [11828.086254] ? __hrtimer_run_queues+0x1c2/0x4d0 [11828.086259] ? watchdog+0x40/0x40 [11828.086263] ? hrtimer_interrupt+0x128/0x2c0 [11828.086271] ? smp_apic_timer_interrupt+0xa6/0x270 [11828.086280] ? apic_timer_interrupt+0xf/0x20 [11828.086285] [11828.086288] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11828.086292] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11828.086297] do_raw_spin_lock+0x177/0x190 [11828.086303] _raw_spin_lock+0x21/0x30 [11828.086309] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11828.086919] obd_mod_free+0x46/0x140 [ptlrpc] [11828.087119] mdc_commit_open+0x91/0xb0 [mdc] [11828.087154] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11828.087509] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11828.087785] after_reply+0xe2c/0x1780 [ptlrpc] [11828.087983] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11828.088669] ? __next_timer_interrupt+0x160/0x160 [11828.088680] ptlrpcd+0xa82/0xe40 [ptlrpc] [11828.088899] ? do_wait_intr+0xf0/0xf0 [11828.088909] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11828.089129] kthread+0x1d1/0x200 [11828.089134] ? set_kthread_struct+0x70/0x70 [11828.089137] ret_from_fork+0x1f/0x30 [11828.309403] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11828.362840] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11828.372032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11828.381870] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11828.386258] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11828.403700] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11828.409942] RAX: 0000000000006092 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11828.415828] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11828.420852] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11828.424537] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11828.431249] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11828.438000] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11828.443571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11828.446339] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11828.450963] Call Trace: [11828.452973] [11828.454983] ? show_regs.cold.9+0x22/0x2f [11828.458808] ? watchdog_timer_fn.cold.6+0x62/0x10a [11828.462950] ? __hrtimer_run_queues+0x1c2/0x4d0 [11828.465711] ? watchdog+0x40/0x40 [11828.471582] ? hrtimer_interrupt+0x128/0x2c0 [11828.475565] ? smp_apic_timer_interrupt+0xa6/0x270 [11828.492726] ? apic_timer_interrupt+0xf/0x20 [11828.495116] [11828.497530] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11828.502153] do_raw_spin_lock+0x177/0x190 [11828.504744] _raw_spin_lock+0x21/0x30 [11828.506708] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11828.510922] ? do_raw_spin_unlock+0x75/0x190 [11828.513814] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11828.517818] ? do_wait_intr+0xf0/0xf0 [11828.520409] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11828.524702] mdc_close+0x53b/0xd70 [mdc] [11828.529879] lmv_close+0x18d/0x490 [lmv] [11828.534383] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11828.537122] ll_md_real_close+0xa5/0x2a0 [lustre] [11828.545453] ll_md_close+0x274/0x800 [lustre] [11828.548003] ll_file_release+0x128/0xc50 [lustre] [11828.558525] __fput+0x10d/0x440 [11828.562353] ____fput+0x12/0x20 [11828.565503] task_work_run+0xc6/0x110 [11828.567906] exit_to_usermode_loop+0x1dd/0x1f0 [11828.570876] do_syscall_64+0x3d6/0x440 [11828.572864] entry_SYSCALL_64_after_hwframe+0x49/0xae [11828.581605] RIP: 0033:0x7f1a63983cd8 [11828.582556] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11828.602700] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11828.611053] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11828.613969] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11828.619100] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11828.621642] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11828.623819] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11832.056134] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11832.060625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11832.087478] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11832.093126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11832.099653] RIP: 0010:native_safe_halt+0x2e/0x50 [11832.101934] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11832.110915] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11832.114743] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11832.116453] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11832.122532] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11832.126447] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11832.128770] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11832.132134] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11832.135673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11832.137203] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11832.142569] Call Trace: [11832.143648] [11832.144645] ? show_regs.cold.9+0x22/0x2f [11832.146017] ? watchdog_timer_fn.cold.6+0x62/0x10a [11832.147960] ? __hrtimer_run_queues+0x1c2/0x4d0 [11832.150427] ? watchdog+0x40/0x40 [11832.151727] ? hrtimer_interrupt+0x128/0x2c0 [11832.154092] ? smp_apic_timer_interrupt+0xa6/0x270 [11832.156735] ? apic_timer_interrupt+0xf/0x20 [11832.158986] [11832.160676] ? native_safe_halt+0x2e/0x50 [11832.162994] kvm_wait+0xcd/0xe0 [11832.165493] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11832.169016] do_raw_spin_lock+0x177/0x190 [11832.170598] _raw_spin_lock+0x21/0x30 [11832.172078] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11832.174575] ? do_raw_spin_unlock+0x75/0x190 [11832.176633] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11832.178840] ? do_wait_intr+0xf0/0xf0 [11832.180746] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11832.182666] mdc_close+0x53b/0xd70 [mdc] [11832.184051] lmv_close+0x18d/0x490 [lmv] [11832.187327] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11832.189956] ll_md_real_close+0xa5/0x2a0 [lustre] [11832.192988] ll_md_close+0x274/0x800 [lustre] [11832.195143] ll_file_release+0x128/0xc50 [lustre] [11832.196712] __fput+0x10d/0x440 [11832.199555] ____fput+0x12/0x20 [11832.200776] task_work_run+0xc6/0x110 [11832.204234] exit_to_usermode_loop+0x1dd/0x1f0 [11832.207741] do_syscall_64+0x3d6/0x440 [11832.208850] entry_SYSCALL_64_after_hwframe+0x49/0xae [11832.211888] RIP: 0033:0x7f1a63983cd8 [11832.212991] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11832.222512] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11832.225995] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11832.228699] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11832.230685] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11832.233312] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11832.235873] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11852.053018] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [11852.062500] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11852.104747] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11852.115940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11852.131553] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11852.140594] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11852.156849] RSP: 0018:ffffc38541c37500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11852.171795] RAX: 0000000000002d16 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11852.176672] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11852.181422] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11852.189459] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11852.192672] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11852.195823] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11852.199505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11852.201779] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11852.204605] Call Trace: [11852.205718] [11852.206616] ? show_regs.cold.9+0x22/0x2f [11852.208276] ? watchdog_timer_fn.cold.6+0x62/0x10a [11852.210267] ? __hrtimer_run_queues+0x1c2/0x4d0 [11852.211805] ? watchdog+0x40/0x40 [11852.212918] ? hrtimer_interrupt+0x128/0x2c0 [11852.214525] ? smp_apic_timer_interrupt+0xa6/0x270 [11852.216514] ? apic_timer_interrupt+0xf/0x20 [11852.218097] [11852.219476] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11852.221516] do_raw_spin_lock+0x177/0x190 [11852.223819] _raw_spin_lock+0x21/0x30 [11852.225377] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11852.227773] ? do_raw_spin_unlock+0x75/0x190 [11852.229340] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11852.231368] ? do_wait_intr+0xf0/0xf0 [11852.233018] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11852.235138] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11852.237307] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11852.240306] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11852.242736] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11852.244942] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11852.247049] mdc_intent_lock+0x353/0x7e0 [mdc] [11852.248616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11852.250999] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11852.253790] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11852.256228] lmv_intent_open+0x338/0x1150 [lmv] [11852.258417] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11852.260776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11852.262994] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11852.264919] ll_intent_lock+0xc5/0xaa0 [lustre] [11852.266715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11852.269089] ll_lookup_it+0xe0e/0x2270 [lustre] [11852.270887] ? do_raw_spin_unlock+0x75/0x190 [11852.272578] ? _raw_spin_unlock+0x12/0x30 [11852.273909] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11852.276126] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11852.278091] ll_atomic_open+0x36a/0xd70 [lustre] [11852.279903] lookup_open+0x356/0xb40 [11852.280873] do_last+0x313/0xfc0 [11852.281714] ? nd_jump_root+0xe5/0x160 [11852.283219] ? path_init+0x437/0x520 [11852.284430] path_openat+0xf7/0x500 [11852.285464] ? do_raw_spin_unlock+0x75/0x190 [11852.287103] do_filp_open+0x99/0x140 [11852.288819] ? getname_flags+0x6e/0x330 [11852.290406] ? __check_object_size+0xff/0x256 [11852.291581] ? do_raw_spin_unlock+0x75/0x190 [11852.293284] ? _raw_spin_unlock+0x12/0x30 [11852.294734] do_sys_openat2+0x2b4/0x410 [11852.295992] do_sys_open+0x73/0xa0 [11852.297371] __x64_sys_openat+0x24/0x30 [11852.298682] do_syscall_64+0xc1/0x440 [11852.300330] entry_SYSCALL_64_after_hwframe+0x49/0xae [11852.301884] RIP: 0033:0x7f1a63983332 [11852.303280] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11852.309495] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11852.312071] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11852.314551] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11852.316791] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11852.319355] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11852.321826] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11856.057021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11856.060404] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11856.060625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11856.060767] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11856.060773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11856.060776] RIP: 0010:native_safe_halt+0x2e/0x50 [11856.060789] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11856.060904] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11856.060909] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11856.060912] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [11856.060924] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11856.060933] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11856.060935] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11856.060938] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11856.060941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11856.060943] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11856.060951] Call Trace: [11856.060956] [11856.060962] ? show_regs.cold.9+0x22/0x2f [11856.060972] ? watchdog_timer_fn.cold.6+0x62/0x10a [11856.060978] ? __hrtimer_run_queues+0x1c2/0x4d0 [11856.060982] ? watchdog+0x40/0x40 [11856.060986] ? hrtimer_interrupt+0x128/0x2c0 [11856.060991] ? smp_apic_timer_interrupt+0xa6/0x270 [11856.060997] ? apic_timer_interrupt+0xf/0x20 [11856.061002] [11856.061006] ? native_safe_halt+0x2e/0x50 [11856.061010] kvm_wait+0xcd/0xe0 [11856.061017] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11856.061026] do_raw_spin_lock+0x177/0x190 [11856.061031] _raw_spin_lock+0x21/0x30 [11856.061036] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11856.061435] ? do_raw_spin_unlock+0x75/0x190 [11856.061441] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11856.061710] ? do_wait_intr+0xf0/0xf0 [11856.061717] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11856.062107] mdc_close+0x53b/0xd70 [mdc] [11856.062153] lmv_close+0x18d/0x490 [lmv] [11856.062311] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11856.062534] ll_md_real_close+0xa5/0x2a0 [lustre] [11856.062619] ll_md_close+0x274/0x800 [lustre] [11856.062702] ll_file_release+0x128/0xc50 [lustre] [11856.062862] __fput+0x10d/0x440 [11856.062873] ____fput+0x12/0x20 [11856.062877] task_work_run+0xc6/0x110 [11856.062884] exit_to_usermode_loop+0x1dd/0x1f0 [11856.062890] do_syscall_64+0x3d6/0x440 [11856.062895] entry_SYSCALL_64_after_hwframe+0x49/0xae [11856.062903] RIP: 0033:0x7f1a63983cd8 [11856.062910] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11856.062915] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11856.062935] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11856.062938] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11856.062940] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11856.062942] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11856.062944] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11856.260894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11856.280873] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11856.285604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11856.289448] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11856.291782] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11856.300243] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11856.305460] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11856.308430] RDX: 000000000000782f RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11856.312750] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11856.315759] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11856.319087] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11856.322099] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11856.325734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11856.328773] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11856.332089] Call Trace: [11856.333122] [11856.334115] ? show_regs.cold.9+0x22/0x2f [11856.335820] ? watchdog_timer_fn.cold.6+0x62/0x10a [11856.337843] ? __hrtimer_run_queues+0x1c2/0x4d0 [11856.339821] ? watchdog+0x40/0x40 [11856.341315] ? hrtimer_interrupt+0x128/0x2c0 [11856.343240] ? smp_apic_timer_interrupt+0xa6/0x270 [11856.345110] ? apic_timer_interrupt+0xf/0x20 [11856.346982] [11856.347840] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11856.350377] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11856.352583] do_raw_spin_lock+0x177/0x190 [11856.354105] _raw_spin_lock+0x21/0x30 [11856.355779] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11856.358346] obd_mod_free+0x46/0x140 [ptlrpc] [11856.360661] mdc_commit_open+0x91/0xb0 [mdc] [11856.362604] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11856.365160] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11856.367725] after_reply+0xe2c/0x1780 [ptlrpc] [11856.369995] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11856.372717] ? __next_timer_interrupt+0x160/0x160 [11856.374806] ptlrpcd+0xa82/0xe40 [ptlrpc] [11856.376910] ? do_wait_intr+0xf0/0xf0 [11856.378705] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11856.380916] kthread+0x1d1/0x200 [11856.382569] ? set_kthread_struct+0x70/0x70 [11856.384395] ret_from_fork+0x1f/0x30 [11856.865323] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9833s! [11856.886420] Showing busy workqueues and worker pools: [11856.897851] workqueue events: flags=0x0 [11856.902907] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11856.914809] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11856.923757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11856.932350] pending: sec_gc_main [ptlrpc] [11856.934336] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11856.952967] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11856.966566] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11856.983907] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11856.999880] workqueue events_long: flags=0x0 [11857.002551] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11857.006442] pending: lu_objects_flush [obdclass] [11857.009594] workqueue events_unbound: flags=0x2 [11857.012483] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11857.017855] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11857.023447] workqueue events_power_efficient: flags=0x80 [11857.025827] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11857.031003] pending: neigh_periodic_work, do_cache_clean [11857.035790] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11857.045573] pending: neigh_periodic_work, check_lifetime [11857.052592] workqueue mm_percpu_wq: flags=0x8 [11857.054234] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11857.065451] pending: vmstat_update [11857.068681] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11857.073050] pending: vmstat_update [11857.077709] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11857.086348] pending: vmstat_update [11857.090369] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11857.094930] pending: vmstat_update [11857.096350] workqueue writeback: flags=0x4a [11857.099830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11857.107400] pending: wb_workfn, wb_workfn [11857.111964] workqueue xprtiod: flags=0xa [11857.116963] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11857.123054] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11857.127729] workqueue ipv6_addrconf: flags=0x40008 [11857.130492] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11857.136829] pending: addrconf_verify_work [11857.139355] workqueue knbd0-recv: flags=0x1a [11857.141424] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11857.144802] in-flight: 647:recv_work [11857.146818] workqueue ptlrpc_pinger: flags=0x2 [11857.149239] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11857.152219] pending: ptlrpc_pinger_main [ptlrpc] [11857.155846] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11847s workers=2 idle: 652 [11860.058371] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11860.065037] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11860.090904] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11860.099421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11860.104421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11860.107966] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11860.121867] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11860.132245] RAX: 0000000000007b0c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11860.136677] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11860.140840] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11860.147180] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11860.150717] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11860.154795] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11860.160000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11860.163320] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11860.166527] Call Trace: [11860.169112] [11860.172545] ? show_regs.cold.9+0x22/0x2f [11860.175788] ? watchdog_timer_fn.cold.6+0x62/0x10a [11860.180798] ? __hrtimer_run_queues+0x1c2/0x4d0 [11860.183940] ? watchdog+0x40/0x40 [11860.187713] ? hrtimer_interrupt+0x128/0x2c0 [11860.193617] ? smp_apic_timer_interrupt+0xa6/0x270 [11860.196229] ? apic_timer_interrupt+0xf/0x20 [11860.198685] [11860.199685] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11860.202533] do_raw_spin_lock+0x177/0x190 [11860.205105] _raw_spin_lock+0x21/0x30 [11860.207674] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11860.212564] ? do_raw_spin_unlock+0x75/0x190 [11860.215644] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11860.220514] ? do_wait_intr+0xf0/0xf0 [11860.222674] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11860.229751] mdc_close+0x53b/0xd70 [mdc] [11860.231869] lmv_close+0x18d/0x490 [lmv] [11860.235300] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11860.243345] ll_md_real_close+0xa5/0x2a0 [lustre] [11860.245841] ll_md_close+0x274/0x800 [lustre] [11860.250255] ll_file_release+0x128/0xc50 [lustre] [11860.252624] __fput+0x10d/0x440 [11860.254358] ____fput+0x12/0x20 [11860.255966] task_work_run+0xc6/0x110 [11860.257733] exit_to_usermode_loop+0x1dd/0x1f0 [11860.259864] do_syscall_64+0x3d6/0x440 [11860.262376] entry_SYSCALL_64_after_hwframe+0x49/0xae [11860.265483] RIP: 0033:0x7f1a63983cd8 [11860.270716] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11860.293219] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11860.303874] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11860.314505] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11860.326708] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11860.331477] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11860.340518] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11884.053023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11884.056828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11884.057019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11884.067558] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11884.067831] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11884.067903] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11884.067910] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11884.067913] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11884.067925] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11884.067929] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11884.067933] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11884.067935] RDX: 0000000000002486 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11884.067937] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11884.067940] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11884.067942] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11884.067945] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11884.067948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11884.067950] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11884.067958] Call Trace: [11884.067967] [11884.067973] ? show_regs.cold.9+0x22/0x2f [11884.067983] ? watchdog_timer_fn.cold.6+0x62/0x10a [11884.067988] ? __hrtimer_run_queues+0x1c2/0x4d0 [11884.067993] ? watchdog+0x40/0x40 [11884.067997] ? hrtimer_interrupt+0x128/0x2c0 [11884.068005] ? smp_apic_timer_interrupt+0xa6/0x270 [11884.068014] ? apic_timer_interrupt+0xf/0x20 [11884.068019] [11884.068022] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11884.068026] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11884.068031] do_raw_spin_lock+0x177/0x190 [11884.068035] _raw_spin_lock+0x21/0x30 [11884.068041] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11884.069216] obd_mod_free+0x46/0x140 [ptlrpc] [11884.070872] mdc_commit_open+0x91/0xb0 [mdc] [11884.070908] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11884.072400] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11884.072832] after_reply+0xe2c/0x1780 [ptlrpc] [11884.074201] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11884.075085] ? __next_timer_interrupt+0x160/0x160 [11884.075099] ptlrpcd+0xa82/0xe40 [ptlrpc] [11884.075487] ? do_wait_intr+0xf0/0xf0 [11884.075498] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11884.078099] kthread+0x1d1/0x200 [11884.078107] ? set_kthread_struct+0x70/0x70 [11884.078111] ret_from_fork+0x1f/0x30 [11884.091103] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11884.091119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11884.091122] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11884.091136] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11884.091141] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11884.091146] RAX: 000000000000320b RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11884.091149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11884.091151] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11884.091153] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11884.091156] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11884.091160] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11884.091163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11884.091166] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11884.091174] Call Trace: [11884.091182] [11884.095364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11884.095445] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11884.095453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11884.095456] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11884.095580] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11884.095591] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11884.095615] RAX: 0000000000001838 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11884.095619] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11884.095621] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11884.095623] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11884.095626] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11884.095629] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11884.095632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11884.095634] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11884.095641] Call Trace: [11884.095648] [11884.095676] ? show_regs.cold.9+0x22/0x2f [11884.095694] ? watchdog_timer_fn.cold.6+0x62/0x10a [11884.095704] ? __hrtimer_run_queues+0x1c2/0x4d0 [11884.095712] ? watchdog+0x40/0x40 [11884.095726] ? hrtimer_interrupt+0x128/0x2c0 [11884.095730] ? smp_apic_timer_interrupt+0xa6/0x270 [11884.095738] ? apic_timer_interrupt+0xf/0x20 [11884.095743] [11884.095747] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11884.095753] do_raw_spin_lock+0x177/0x190 [11884.095758] _raw_spin_lock+0x21/0x30 [11884.095767] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11884.096152] ? do_raw_spin_unlock+0x75/0x190 [11884.096158] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11884.096385] ? do_wait_intr+0xf0/0xf0 [11884.096394] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11884.096625] mdc_close+0x53b/0xd70 [mdc] [11884.096699] lmv_close+0x18d/0x490 [lmv] [11884.096735] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11884.097124] ll_md_real_close+0xa5/0x2a0 [lustre] [11884.097226] ll_md_close+0x274/0x800 [lustre] [11884.097338] ll_file_release+0x128/0xc50 [lustre] [11884.097424] __fput+0x10d/0x440 [11884.097434] ____fput+0x12/0x20 [11884.097438] task_work_run+0xc6/0x110 [11884.097447] exit_to_usermode_loop+0x1dd/0x1f0 [11884.097460] do_syscall_64+0x3d6/0x440 [11884.097465] entry_SYSCALL_64_after_hwframe+0x49/0xae [11884.097471] RIP: 0033:0x7f1a63983cd8 [11884.097478] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11884.097482] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11884.097487] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11884.097490] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11884.097492] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11884.097494] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11884.097496] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11884.485553] ? show_regs.cold.9+0x22/0x2f [11884.487837] ? watchdog_timer_fn.cold.6+0x62/0x10a [11884.490166] ? __hrtimer_run_queues+0x1c2/0x4d0 [11884.492787] ? watchdog+0x40/0x40 [11884.494907] ? hrtimer_interrupt+0x128/0x2c0 [11884.497946] ? smp_apic_timer_interrupt+0xa6/0x270 [11884.500852] ? apic_timer_interrupt+0xf/0x20 [11884.504893] [11884.505673] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11884.507623] do_raw_spin_lock+0x177/0x190 [11884.509033] _raw_spin_lock+0x21/0x30 [11884.510357] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11884.512465] ? do_raw_spin_unlock+0x75/0x190 [11884.514356] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11884.516798] ? do_wait_intr+0xf0/0xf0 [11884.518350] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11884.520319] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11884.522469] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11884.524951] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11884.527252] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11884.529182] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11884.530835] mdc_intent_lock+0x353/0x7e0 [mdc] [11884.532520] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11884.534948] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11884.537195] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11884.539161] lmv_intent_open+0x338/0x1150 [lmv] [11884.541399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11884.543577] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11884.549892] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11884.551735] ll_intent_lock+0xc5/0xaa0 [lustre] [11884.553900] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11884.556488] ll_lookup_it+0xe0e/0x2270 [lustre] [11884.558330] ? do_raw_spin_unlock+0x75/0x190 [11884.560216] ? _raw_spin_unlock+0x12/0x30 [11884.561939] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11884.565748] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11884.570190] ll_atomic_open+0x36a/0xd70 [lustre] [11884.572917] lookup_open+0x356/0xb40 [11884.574986] do_last+0x313/0xfc0 [11884.576529] ? nd_jump_root+0xe5/0x160 [11884.579013] ? path_init+0x437/0x520 [11884.580590] path_openat+0xf7/0x500 [11884.582242] ? do_raw_spin_unlock+0x75/0x190 [11884.584449] do_filp_open+0x99/0x140 [11884.586916] ? getname_flags+0x6e/0x330 [11884.588375] ? __check_object_size+0xff/0x256 [11884.590434] ? do_raw_spin_unlock+0x75/0x190 [11884.592342] ? _raw_spin_unlock+0x12/0x30 [11884.594932] do_sys_openat2+0x2b4/0x410 [11884.596816] do_sys_open+0x73/0xa0 [11884.598182] __x64_sys_openat+0x24/0x30 [11884.599905] do_syscall_64+0xc1/0x440 [11884.601844] entry_SYSCALL_64_after_hwframe+0x49/0xae [11884.604162] RIP: 0033:0x7f1a63983332 [11884.606168] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11884.616328] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11884.620373] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11884.623869] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11884.627846] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11884.631223] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11884.635131] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11887.585108] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9864s! [11887.588567] Showing busy workqueues and worker pools: [11887.591019] workqueue events: flags=0x0 [11887.592885] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11887.596053] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11887.599495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11887.602669] pending: sec_gc_main [ptlrpc] [11887.604539] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11887.607491] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11887.612380] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11887.615134] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11887.619710] workqueue events_long: flags=0x0 [11887.621708] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11887.624826] pending: lu_objects_flush [obdclass] [11887.627353] workqueue events_unbound: flags=0x2 [11887.629538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11887.632537] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11887.637282] workqueue events_power_efficient: flags=0x80 [11887.640143] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11887.642412] pending: neigh_periodic_work, do_cache_clean [11887.644678] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11887.647936] pending: neigh_periodic_work, check_lifetime [11887.650628] workqueue mm_percpu_wq: flags=0x8 [11887.652758] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11887.655893] pending: vmstat_update [11887.657532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11887.661032] pending: vmstat_update [11887.662096] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11887.665023] pending: vmstat_update [11887.666676] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11887.669911] pending: vmstat_update [11887.671479] workqueue writeback: flags=0x4a [11887.673523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11887.675482] pending: wb_workfn, wb_workfn [11887.676788] workqueue xprtiod: flags=0xa [11887.678009] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11887.680513] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11887.682904] workqueue ipv6_addrconf: flags=0x40008 [11887.684439] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11887.686401] pending: addrconf_verify_work [11887.687494] workqueue knbd0-recv: flags=0x1a [11887.688720] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11887.690855] in-flight: 647:recv_work [11887.691893] workqueue ptlrpc_pinger: flags=0x2 [11887.693301] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11887.694840] pending: ptlrpc_pinger_main [ptlrpc] [11887.696501] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11877s workers=2 idle: 652 [11888.056638] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11888.064053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11888.086591] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11888.095421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11888.103949] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11888.109868] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11888.130945] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11888.135200] RAX: 0000000000005886 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11888.147106] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11888.156089] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11888.160987] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11888.167827] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11888.172125] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11888.178453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11888.183087] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11888.186498] Call Trace: [11888.189535] [11888.190262] ? show_regs.cold.9+0x22/0x2f [11888.191608] ? watchdog_timer_fn.cold.6+0x62/0x10a [11888.193804] ? __hrtimer_run_queues+0x1c2/0x4d0 [11888.195292] ? watchdog+0x40/0x40 [11888.196333] ? hrtimer_interrupt+0x128/0x2c0 [11888.200292] ? smp_apic_timer_interrupt+0xa6/0x270 [11888.205430] ? apic_timer_interrupt+0xf/0x20 [11888.207198] [11888.208274] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11888.211962] do_raw_spin_lock+0x177/0x190 [11888.213419] _raw_spin_lock+0x21/0x30 [11888.215851] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11888.218063] ? do_raw_spin_unlock+0x75/0x190 [11888.219484] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11888.229956] ? do_wait_intr+0xf0/0xf0 [11888.231092] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11888.238197] mdc_close+0x53b/0xd70 [mdc] [11888.239297] lmv_close+0x18d/0x490 [lmv] [11888.240383] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11888.244901] ll_md_real_close+0xa5/0x2a0 [lustre] [11888.246473] ll_md_close+0x274/0x800 [lustre] [11888.250939] ll_file_release+0x128/0xc50 [lustre] [11888.252980] __fput+0x10d/0x440 [11888.256253] ____fput+0x12/0x20 [11888.257281] task_work_run+0xc6/0x110 [11888.262535] exit_to_usermode_loop+0x1dd/0x1f0 [11888.267219] do_syscall_64+0x3d6/0x440 [11888.268401] entry_SYSCALL_64_after_hwframe+0x49/0xae [11888.274186] RIP: 0033:0x7f1a63983cd8 [11888.275792] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11888.285274] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11888.293207] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11888.295797] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11888.298840] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11888.301917] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11888.305234] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11912.053089] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11912.056095] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11912.057027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [11912.065352] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [11912.065368] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11912.065458] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11912.065472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11912.065474] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11912.065487] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11912.065492] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11912.065496] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11912.065499] RDX: 000000000000365a RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11912.065501] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11912.065503] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11912.065506] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11912.065510] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11912.065517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11912.065520] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11912.065527] Call Trace: [11912.065532] [11912.065538] ? show_regs.cold.9+0x22/0x2f [11912.065548] ? watchdog_timer_fn.cold.6+0x62/0x10a [11912.065554] ? __hrtimer_run_queues+0x1c2/0x4d0 [11912.065559] ? watchdog+0x40/0x40 [11912.065563] ? hrtimer_interrupt+0x128/0x2c0 [11912.065567] ? smp_apic_timer_interrupt+0xa6/0x270 [11912.065575] ? apic_timer_interrupt+0xf/0x20 [11912.065580] [11912.065583] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11912.065587] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11912.065591] do_raw_spin_lock+0x177/0x190 [11912.065595] _raw_spin_lock+0x21/0x30 [11912.065600] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11912.066411] obd_mod_free+0x46/0x140 [ptlrpc] [11912.066639] mdc_commit_open+0x91/0xb0 [mdc] [11912.066672] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11912.066942] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11912.067265] after_reply+0xe2c/0x1780 [ptlrpc] [11912.068319] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11912.068662] ? __next_timer_interrupt+0x160/0x160 [11912.068672] ptlrpcd+0xa82/0xe40 [ptlrpc] [11912.068893] ? do_wait_intr+0xf0/0xf0 [11912.068903] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11912.069261] kthread+0x1d1/0x200 [11912.069268] ? set_kthread_struct+0x70/0x70 [11912.069272] ret_from_fork+0x1f/0x30 [11912.083929] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11912.083945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11912.083948] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11912.083961] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11912.083966] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11912.083972] RAX: 0000000000007b81 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11912.083974] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11912.083976] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11912.083979] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11912.083981] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11912.083984] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11912.083987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11912.083990] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11912.083999] Call Trace: [11912.084004] [11912.084009] ? show_regs.cold.9+0x22/0x2f [11912.084019] ? watchdog_timer_fn.cold.6+0x62/0x10a [11912.084025] ? __hrtimer_run_queues+0x1c2/0x4d0 [11912.084030] ? watchdog+0x40/0x40 [11912.084034] ? hrtimer_interrupt+0x128/0x2c0 [11912.084037] ? smp_apic_timer_interrupt+0xa6/0x270 [11912.084045] ? apic_timer_interrupt+0xf/0x20 [11912.084050] [11912.084053] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11912.084060] do_raw_spin_lock+0x177/0x190 [11912.084065] _raw_spin_lock+0x21/0x30 [11912.084070] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11912.084326] ? do_raw_spin_unlock+0x75/0x190 [11912.084332] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11912.084880] ? do_wait_intr+0xf0/0xf0 [11912.084890] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11912.085083] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11912.085298] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11912.085591] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11912.085697] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11912.085734] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11912.085764] mdc_intent_lock+0x353/0x7e0 [mdc] [11912.085793] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11912.085878] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11912.086089] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11912.086119] lmv_intent_open+0x338/0x1150 [lmv] [11912.086150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11912.086233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11912.086315] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11912.086337] ll_intent_lock+0xc5/0xaa0 [lustre] [11912.086538] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11912.086617] ll_lookup_it+0xe0e/0x2270 [lustre] [11912.092178] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11912.092836] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11912.092843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11912.092846] RIP: 0010:native_safe_halt+0x2e/0x50 [11912.092866] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11912.092870] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11912.092874] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11912.092876] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11912.092878] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11912.092880] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11912.092882] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11912.092885] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11912.092894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11912.092896] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11912.092905] Call Trace: [11912.093141] [11912.093148] ? show_regs.cold.9+0x22/0x2f [11912.093159] ? watchdog_timer_fn.cold.6+0x62/0x10a [11912.093165] ? __hrtimer_run_queues+0x1c2/0x4d0 [11912.093170] ? watchdog+0x40/0x40 [11912.093174] ? hrtimer_interrupt+0x128/0x2c0 [11912.093178] ? smp_apic_timer_interrupt+0xa6/0x270 [11912.093186] ? apic_timer_interrupt+0xf/0x20 [11912.093191] [11912.093194] ? native_safe_halt+0x2e/0x50 [11912.093199] kvm_wait+0xcd/0xe0 [11912.093796] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11912.094706] do_raw_spin_lock+0x177/0x190 [11912.094712] _raw_spin_lock+0x21/0x30 [11912.094718] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11912.100188] ? do_raw_spin_unlock+0x75/0x190 [11912.136147] ? do_raw_spin_unlock+0x75/0x190 [11912.136169] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11912.137675] ? do_wait_intr+0xf0/0xf0 [11912.137693] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11912.138211] mdc_close+0x53b/0xd70 [mdc] [11912.138540] lmv_close+0x18d/0x490 [lmv] [11912.138583] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11912.138691] ll_md_real_close+0xa5/0x2a0 [lustre] [11912.138770] ll_md_close+0x274/0x800 [lustre] [11912.138851] ll_file_release+0x128/0xc50 [lustre] [11912.138934] __fput+0x10d/0x440 [11912.138944] ____fput+0x12/0x20 [11912.138948] task_work_run+0xc6/0x110 [11912.138957] exit_to_usermode_loop+0x1dd/0x1f0 [11912.138966] do_syscall_64+0x3d6/0x440 [11912.138971] entry_SYSCALL_64_after_hwframe+0x49/0xae [11912.138980] RIP: 0033:0x7f1a63983cd8 [11912.138987] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11912.138991] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11912.138997] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11912.138999] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11912.139002] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11912.139004] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11912.139006] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11912.813264] ? _raw_spin_unlock+0x12/0x30 [11912.815767] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11912.827559] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11912.833328] ll_atomic_open+0x36a/0xd70 [lustre] [11912.837020] lookup_open+0x356/0xb40 [11912.840084] do_last+0x313/0xfc0 [11912.842126] ? nd_jump_root+0xe5/0x160 [11912.846043] ? path_init+0x437/0x520 [11912.848315] path_openat+0xf7/0x500 [11912.852306] ? do_raw_spin_unlock+0x75/0x190 [11912.854756] do_filp_open+0x99/0x140 [11912.855998] ? getname_flags+0x6e/0x330 [11912.865016] ? __check_object_size+0xff/0x256 [11912.867906] ? do_raw_spin_unlock+0x75/0x190 [11912.875063] ? _raw_spin_unlock+0x12/0x30 [11912.877338] do_sys_openat2+0x2b4/0x410 [11912.884557] do_sys_open+0x73/0xa0 [11912.886361] __x64_sys_openat+0x24/0x30 [11912.888502] do_syscall_64+0xc1/0x440 [11912.889612] entry_SYSCALL_64_after_hwframe+0x49/0xae [11912.893678] RIP: 0033:0x7f1a63983332 [11912.896184] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11912.907879] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11912.911025] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11912.916253] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11912.920843] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11912.923602] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11912.929088] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11916.056014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11916.060167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11916.085879] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11916.091397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11916.097066] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11916.100274] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11916.110827] RSP: 0018:ffffc38542073af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11916.114058] RAX: 0000000000006f8c RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11916.117453] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11916.120783] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11916.123618] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11916.126718] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11916.129967] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11916.134124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11916.136754] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11916.142401] Call Trace: [11916.143574] [11916.144324] ? show_regs.cold.9+0x22/0x2f [11916.146065] ? watchdog_timer_fn.cold.6+0x62/0x10a [11916.148923] ? __hrtimer_run_queues+0x1c2/0x4d0 [11916.151104] ? watchdog+0x40/0x40 [11916.152482] ? hrtimer_interrupt+0x128/0x2c0 [11916.154275] ? smp_apic_timer_interrupt+0xa6/0x270 [11916.156320] ? apic_timer_interrupt+0xf/0x20 [11916.158884] [11916.159939] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11916.164080] do_raw_spin_lock+0x177/0x190 [11916.167086] _raw_spin_lock+0x21/0x30 [11916.168802] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11916.177860] ? do_raw_spin_unlock+0x75/0x190 [11916.182420] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11916.187514] ? do_wait_intr+0xf0/0xf0 [11916.191459] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11916.195355] mdc_close+0x53b/0xd70 [mdc] [11916.196862] lmv_close+0x18d/0x490 [lmv] [11916.198808] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11916.201811] ll_md_real_close+0xa5/0x2a0 [lustre] [11916.204278] ll_md_close+0x274/0x800 [lustre] [11916.206754] ll_file_release+0x128/0xc50 [lustre] [11916.209115] __fput+0x10d/0x440 [11916.210511] ____fput+0x12/0x20 [11916.211875] task_work_run+0xc6/0x110 [11916.213836] exit_to_usermode_loop+0x1dd/0x1f0 [11916.215628] do_syscall_64+0x3d6/0x440 [11916.217397] entry_SYSCALL_64_after_hwframe+0x49/0xae [11916.219368] RIP: 0033:0x7f1a63983cd8 [11916.221741] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11916.236404] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11916.245469] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11916.251792] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11916.257080] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11916.261524] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11916.273783] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11917.792078] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9894s! [11917.795133] Showing busy workqueues and worker pools: [11917.797458] workqueue events: flags=0x0 [11917.799030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11917.801992] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11917.805287] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11917.808266] pending: sec_gc_main [ptlrpc] [11917.810874] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11917.813584] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11917.821504] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11917.831187] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11917.844564] workqueue events_long: flags=0x0 [11917.846884] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11917.850154] pending: lu_objects_flush [obdclass] [11917.852604] workqueue events_unbound: flags=0x2 [11917.854509] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11917.856743] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11917.861736] workqueue events_power_efficient: flags=0x80 [11917.864796] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11917.867418] pending: neigh_periodic_work, do_cache_clean [11917.870065] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11917.872889] pending: neigh_periodic_work, check_lifetime [11917.875215] workqueue mm_percpu_wq: flags=0x8 [11917.877850] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11917.881041] pending: vmstat_update [11917.882620] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11917.885157] pending: vmstat_update [11917.886725] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11917.889953] pending: vmstat_update [11917.905437] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11917.913469] pending: vmstat_update [11917.915183] workqueue writeback: flags=0x4a [11917.918162] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11917.925647] pending: wb_workfn, wb_workfn [11917.933539] workqueue xprtiod: flags=0xa [11917.936848] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11917.946855] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11917.954491] workqueue ipv6_addrconf: flags=0x40008 [11917.956779] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11917.960497] pending: addrconf_verify_work [11917.969158] workqueue knbd0-recv: flags=0x1a [11917.971619] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11917.983210] in-flight: 647:recv_work [11917.989903] workqueue ptlrpc_pinger: flags=0x2 [11917.993511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11917.999363] pending: ptlrpc_pinger_main [ptlrpc] [11918.005722] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11908s workers=2 idle: 652 [11940.057841] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11940.058018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [11940.062356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11940.062440] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11940.062448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11940.062451] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [11940.062467] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [11940.062470] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11940.062475] RAX: 000000000000346e RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11940.062477] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11940.062479] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11940.062481] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11940.062483] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11940.062486] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11940.062489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11940.062491] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11940.062500] Call Trace: [11940.062507] [11940.062514] ? show_regs.cold.9+0x22/0x2f [11940.067100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [11940.104789] ? watchdog_timer_fn.cold.6+0x62/0x10a [11940.104815] ? __hrtimer_run_queues+0x1c2/0x4d0 [11940.110793] sha512_generic [11940.110809] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11940.110816] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11940.110819] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11940.110834] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11940.110838] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11940.110843] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11940.110845] RDX: 0000000000001678 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11940.110848] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11940.110850] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11940.110853] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11940.110856] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11940.110859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11940.110862] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11940.110871] Call Trace: [11940.110876] [11940.110882] ? show_regs.cold.9+0x22/0x2f [11940.110892] ? watchdog_timer_fn.cold.6+0x62/0x10a [11940.110899] ? __hrtimer_run_queues+0x1c2/0x4d0 [11940.110904] ? watchdog+0x40/0x40 [11940.110908] ? hrtimer_interrupt+0x128/0x2c0 [11940.110911] ? smp_apic_timer_interrupt+0xa6/0x270 [11940.110920] ? apic_timer_interrupt+0xf/0x20 [11940.110925] [11940.110928] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11940.110932] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11940.110937] do_raw_spin_lock+0x177/0x190 [11940.110942] _raw_spin_lock+0x21/0x30 [11940.110950] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11940.111252] obd_mod_free+0x46/0x140 [ptlrpc] [11940.118497] ? watchdog+0x40/0x40 [11940.121886] mdc_commit_open+0x91/0xb0 [mdc] [11940.121936] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11940.136676] ? hrtimer_interrupt+0x128/0x2c0 [11940.140320] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11940.143793] ? smp_apic_timer_interrupt+0xa6/0x270 [11940.143814] ? apic_timer_interrupt+0xf/0x20 [11940.143820] [11940.143825] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [11940.143834] do_raw_spin_lock+0x177/0x190 [11940.143845] _raw_spin_lock+0x21/0x30 [11940.143851] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11940.148039] after_reply+0xe2c/0x1780 [ptlrpc] [11940.152150] ? do_raw_spin_unlock+0x75/0x190 [11940.155116] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11940.159532] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11940.164063] ? __next_timer_interrupt+0x160/0x160 [11940.166573] ? do_wait_intr+0xf0/0xf0 [11940.166594] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11940.170588] ptlrpcd+0xa82/0xe40 [ptlrpc] [11940.171867] mdc_close+0x53b/0xd70 [mdc] [11940.172930] ? do_wait_intr+0xf0/0xf0 [11940.175070] lmv_close+0x18d/0x490 [lmv] [11940.197885] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11940.199868] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11940.202193] kthread+0x1d1/0x200 [11940.202219] ? set_kthread_struct+0x70/0x70 [11940.203497] ll_md_real_close+0xa5/0x2a0 [lustre] [11940.203774] ll_md_close+0x274/0x800 [lustre] [11940.203921] ll_file_release+0x128/0xc50 [lustre] [11940.209995] ret_from_fork+0x1f/0x30 [11940.214357] __fput+0x10d/0x440 [11940.214382] ____fput+0x12/0x20 [11940.214387] task_work_run+0xc6/0x110 [11940.214396] exit_to_usermode_loop+0x1dd/0x1f0 [11940.214412] do_syscall_64+0x3d6/0x440 [11940.214417] entry_SYSCALL_64_after_hwframe+0x49/0xae [11940.214426] RIP: 0033:0x7f1a63983cd8 [11940.388832] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11940.403356] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11940.406880] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11940.410835] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11940.416231] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11940.423886] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11940.432889] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11944.053027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11944.056456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11944.057783] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11944.077682] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11944.081249] Modules linked in: lustre(O) osp(O) [11944.085673] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11944.085686] ofd(O) lod(O) mdt(O) [11944.087807] RIP: 0010:native_safe_halt+0x2e/0x50 [11944.092254] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [11944.093775] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11944.093785] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11944.093791] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11944.093797] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11944.093800] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11944.093802] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11944.093805] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11944.093808] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11944.093811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11944.093814] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11944.093825] Call Trace: [11944.093830] [11944.093836] ? show_regs.cold.9+0x22/0x2f [11944.093854] ? watchdog_timer_fn.cold.6+0x62/0x10a [11944.093861] ? __hrtimer_run_queues+0x1c2/0x4d0 [11944.093866] ? watchdog+0x40/0x40 [11944.093870] ? hrtimer_interrupt+0x128/0x2c0 [11944.093878] ? smp_apic_timer_interrupt+0xa6/0x270 [11944.093887] ? apic_timer_interrupt+0xf/0x20 [11944.093892] [11944.093895] ? native_safe_halt+0x2e/0x50 [11944.093903] kvm_wait+0xcd/0xe0 [11944.093911] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11944.093919] do_raw_spin_lock+0x177/0x190 [11944.093927] _raw_spin_lock+0x21/0x30 [11944.093931] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11944.094271] ? do_raw_spin_unlock+0x75/0x190 [11944.094278] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11944.094500] ? do_wait_intr+0xf0/0xf0 [11944.094508] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11944.094729] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11944.095136] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11944.095475] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11944.095616] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11944.095667] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11944.095696] mdc_intent_lock+0x353/0x7e0 [mdc] [11944.095726] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11944.095831] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11944.096246] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11944.096282] lmv_intent_open+0x338/0x1150 [lmv] [11944.096381] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11944.096655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11944.096765] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11944.096788] ll_intent_lock+0xc5/0xaa0 [lustre] [11944.096885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11944.096981] ll_lookup_it+0xe0e/0x2270 [lustre] [11944.097182] ? do_raw_spin_unlock+0x75/0x190 [11944.097189] ? _raw_spin_unlock+0x12/0x30 [11944.097195] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11944.097217] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11944.097241] ll_atomic_open+0x36a/0xd70 [lustre] [11944.097406] lookup_open+0x356/0xb40 [11944.097414] do_last+0x313/0xfc0 [11944.097419] ? nd_jump_root+0xe5/0x160 [11944.097424] ? path_init+0x437/0x520 [11944.097428] path_openat+0xf7/0x500 [11944.097436] ? do_raw_spin_unlock+0x75/0x190 [11944.097441] do_filp_open+0x99/0x140 [11944.097448] ? getname_flags+0x6e/0x330 [11944.097452] ? __check_object_size+0xff/0x256 [11944.097459] ? do_raw_spin_unlock+0x75/0x190 [11944.097463] ? _raw_spin_unlock+0x12/0x30 [11944.097468] do_sys_openat2+0x2b4/0x410 [11944.097475] do_sys_open+0x73/0xa0 [11944.097479] __x64_sys_openat+0x24/0x30 [11944.097482] do_syscall_64+0xc1/0x440 [11944.097489] entry_SYSCALL_64_after_hwframe+0x49/0xae [11944.097494] RIP: 0033:0x7f1a63983332 [11944.097502] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11944.097506] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11944.097511] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11944.097514] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11944.097517] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11944.097519] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11944.097522] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11944.440646] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11944.449599] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11944.455825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11944.463096] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11944.468572] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11944.478195] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11944.482374] RAX: 0000000000006f50 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11944.485814] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11944.489163] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11944.492450] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11944.498015] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11944.504537] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11944.508578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11944.510855] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11944.516589] Call Trace: [11944.518226] [11944.519433] ? show_regs.cold.9+0x22/0x2f [11944.522379] ? watchdog_timer_fn.cold.6+0x62/0x10a [11944.525739] ? __hrtimer_run_queues+0x1c2/0x4d0 [11944.528125] ? watchdog+0x40/0x40 [11944.529920] ? hrtimer_interrupt+0x128/0x2c0 [11944.534752] ? smp_apic_timer_interrupt+0xa6/0x270 [11944.542425] ? apic_timer_interrupt+0xf/0x20 [11944.545602] [11944.546840] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11944.550716] do_raw_spin_lock+0x177/0x190 [11944.553960] _raw_spin_lock+0x21/0x30 [11944.556171] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11944.559289] ? do_raw_spin_unlock+0x75/0x190 [11944.563769] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11944.573401] ? do_wait_intr+0xf0/0xf0 [11944.577474] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11944.583772] mdc_close+0x53b/0xd70 [mdc] [11944.588052] lmv_close+0x18d/0x490 [lmv] [11944.590694] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11944.596945] ll_md_real_close+0xa5/0x2a0 [lustre] [11944.602508] ll_md_close+0x274/0x800 [lustre] [11944.605156] ll_file_release+0x128/0xc50 [lustre] [11944.607558] __fput+0x10d/0x440 [11944.608853] ____fput+0x12/0x20 [11944.610192] task_work_run+0xc6/0x110 [11944.611870] exit_to_usermode_loop+0x1dd/0x1f0 [11944.616647] do_syscall_64+0x3d6/0x440 [11944.618934] entry_SYSCALL_64_after_hwframe+0x49/0xae [11944.624693] RIP: 0033:0x7f1a63983cd8 [11944.632604] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11944.645939] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11944.655641] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11944.663282] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11944.669821] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11944.674780] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11944.681859] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11948.512282] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9925s! [11948.518631] Showing busy workqueues and worker pools: [11948.528412] workqueue events: flags=0x0 [11948.535728] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11948.548390] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11948.561310] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11948.574329] pending: sec_gc_main [ptlrpc] [11948.589644] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11948.597842] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11948.618556] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11948.627700] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11948.642794] workqueue events_long: flags=0x0 [11948.651285] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11948.661553] pending: lu_objects_flush [obdclass] [11948.665865] workqueue events_unbound: flags=0x2 [11948.673564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11948.679338] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11948.687582] workqueue events_power_efficient: flags=0x80 [11948.689826] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11948.695722] pending: neigh_periodic_work, do_cache_clean [11948.699284] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11948.703589] pending: neigh_periodic_work, check_lifetime [11948.706812] workqueue mm_percpu_wq: flags=0x8 [11948.710169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11948.715401] pending: vmstat_update [11948.717229] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11948.722864] pending: vmstat_update [11948.728856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11948.731186] pending: vmstat_update [11948.733349] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11948.737910] pending: vmstat_update [11948.744591] workqueue writeback: flags=0x4a [11948.747674] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11948.755570] pending: wb_workfn, wb_workfn [11948.757392] workqueue xprtiod: flags=0xa [11948.759483] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11948.762312] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11948.767275] workqueue ipv6_addrconf: flags=0x40008 [11948.770316] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11948.773712] pending: addrconf_verify_work [11948.779428] workqueue knbd0-recv: flags=0x1a [11948.784597] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11948.789688] in-flight: 647:recv_work [11948.793527] workqueue ptlrpc_pinger: flags=0x2 [11948.795204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11948.810357] pending: ptlrpc_pinger_main [ptlrpc] [11948.814646] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11938s workers=2 idle: 652 [11968.057128] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11968.061989] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11968.071521] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_rcv:2385] [11968.107667] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11968.107681] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11968.107684] RIP: 0010:native_safe_halt+0x2e/0x50 [11968.107697] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11968.107701] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11968.107707] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11968.107709] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [11968.107711] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11968.107713] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11968.107715] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11968.107719] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11968.107721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11968.107724] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11968.107742] Call Trace: [11968.107747] [11968.107753] ? show_regs.cold.9+0x22/0x2f [11968.107763] ? watchdog_timer_fn.cold.6+0x62/0x10a [11968.107770] ? __hrtimer_run_queues+0x1c2/0x4d0 [11968.107774] ? watchdog+0x40/0x40 [11968.107778] ? hrtimer_interrupt+0x128/0x2c0 [11968.107781] ? smp_apic_timer_interrupt+0xa6/0x270 [11968.107787] ? apic_timer_interrupt+0xf/0x20 [11968.107791] [11968.107794] ? native_safe_halt+0x2e/0x50 [11968.107797] kvm_wait+0xcd/0xe0 [11968.107804] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11968.107813] do_raw_spin_lock+0x177/0x190 [11968.107817] _raw_spin_lock+0x21/0x30 [11968.107821] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11968.108244] ? do_raw_spin_unlock+0x75/0x190 [11968.108250] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11968.108473] ? do_wait_intr+0xf0/0xf0 [11968.108481] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11968.108672] mdc_close+0x53b/0xd70 [mdc] [11968.108711] lmv_close+0x18d/0x490 [lmv] [11968.108752] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11968.114809] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [11968.126836] ll_md_real_close+0xa5/0x2a0 [lustre] [11968.145500] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11968.145561] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11968.145569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11968.145572] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11968.145586] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11968.145590] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11968.145594] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [11968.145597] RDX: 00000000000076eb RSI: 0000000000000003 RDI: ffff9f8452a252e0 [11968.145599] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [11968.145602] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [11968.145605] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11968.145608] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [11968.145611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11968.145614] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [11968.145622] Call Trace: [11968.145628] [11968.145633] ? show_regs.cold.9+0x22/0x2f [11968.145643] ? watchdog_timer_fn.cold.6+0x62/0x10a [11968.145649] ? __hrtimer_run_queues+0x1c2/0x4d0 [11968.145654] ? watchdog+0x40/0x40 [11968.145658] ? hrtimer_interrupt+0x128/0x2c0 [11968.145663] ? smp_apic_timer_interrupt+0xa6/0x270 [11968.145671] ? apic_timer_interrupt+0xf/0x20 [11968.145676] [11968.145679] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11968.145685] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11968.145690] do_raw_spin_lock+0x177/0x190 [11968.145695] _raw_spin_lock+0x21/0x30 [11968.145701] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11968.147234] obd_mod_free+0x46/0x140 [ptlrpc] [11968.147623] mdc_commit_open+0x91/0xb0 [mdc] [11968.147664] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11968.148265] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11968.148456] after_reply+0xe2c/0x1780 [ptlrpc] [11968.148648] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11968.149233] ? __next_timer_interrupt+0x160/0x160 [11968.149245] ptlrpcd+0xa82/0xe40 [ptlrpc] [11968.149469] ? do_wait_intr+0xf0/0xf0 [11968.149479] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11968.149700] kthread+0x1d1/0x200 [11968.149706] ? set_kthread_struct+0x70/0x70 [11968.149710] ret_from_fork+0x1f/0x30 [11968.552992] ll_md_close+0x274/0x800 [lustre] [11968.556790] ll_file_release+0x128/0xc50 [lustre] [11968.558641] __fput+0x10d/0x440 [11968.559900] ____fput+0x12/0x20 [11968.562945] task_work_run+0xc6/0x110 [11968.565485] exit_to_usermode_loop+0x1dd/0x1f0 [11968.567307] do_syscall_64+0x3d6/0x440 [11968.569761] entry_SYSCALL_64_after_hwframe+0x49/0xae [11968.580822] RIP: 0033:0x7f1a63983cd8 [11968.586399] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11968.601762] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11968.606886] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11968.614715] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11968.619396] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11968.630573] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11968.638211] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11972.056078] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [11972.059303] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11972.103397] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11972.110961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11972.123514] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11972.127096] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11972.141757] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11972.151270] RAX: 000000000000788f RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11972.155901] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [11972.162739] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [11972.168239] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [11972.174740] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [11972.178707] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [11972.185281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11972.198655] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [11972.206880] Call Trace: [11972.208650] [11972.209723] ? show_regs.cold.9+0x22/0x2f [11972.216682] ? watchdog_timer_fn.cold.6+0x62/0x10a [11972.220497] ? __hrtimer_run_queues+0x1c2/0x4d0 [11972.223049] ? watchdog+0x40/0x40 [11972.224781] ? hrtimer_interrupt+0x128/0x2c0 [11972.226652] ? smp_apic_timer_interrupt+0xa6/0x270 [11972.230515] ? apic_timer_interrupt+0xf/0x20 [11972.237909] [11972.239426] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11972.242695] do_raw_spin_lock+0x177/0x190 [11972.250227] _raw_spin_lock+0x21/0x30 [11972.251605] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11972.257353] ? do_raw_spin_unlock+0x75/0x190 [11972.260339] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11972.266566] ? do_wait_intr+0xf0/0xf0 [11972.271496] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11972.276200] mdc_close+0x53b/0xd70 [mdc] [11972.279007] lmv_close+0x18d/0x490 [lmv] [11972.285044] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11972.292256] ll_md_real_close+0xa5/0x2a0 [lustre] [11972.297481] ll_md_close+0x274/0x800 [lustre] [11972.299223] ll_file_release+0x128/0xc50 [lustre] [11972.312538] __fput+0x10d/0x440 [11972.314554] ____fput+0x12/0x20 [11972.318153] task_work_run+0xc6/0x110 [11972.320593] exit_to_usermode_loop+0x1dd/0x1f0 [11972.324359] do_syscall_64+0x3d6/0x440 [11972.326581] entry_SYSCALL_64_after_hwframe+0x49/0xae [11972.329473] RIP: 0033:0x7f1a63983cd8 [11972.332860] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11972.353630] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11972.364010] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [11972.368012] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11972.378781] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [11972.384330] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [11972.391429] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11976.053071] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [11976.063165] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11976.100646] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11976.106375] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11976.110716] RIP: 0010:native_safe_halt+0x2e/0x50 [11976.113116] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11976.124244] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11976.133511] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11976.145200] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [11976.156514] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [11976.167399] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [11976.177302] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [11976.185065] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [11976.192432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11976.197530] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [11976.203795] Call Trace: [11976.204654] [11976.205372] ? show_regs.cold.9+0x22/0x2f [11976.213201] ? watchdog_timer_fn.cold.6+0x62/0x10a [11976.217407] ? __hrtimer_run_queues+0x1c2/0x4d0 [11976.222637] ? watchdog+0x40/0x40 [11976.224523] ? hrtimer_interrupt+0x128/0x2c0 [11976.227693] ? smp_apic_timer_interrupt+0xa6/0x270 [11976.231535] ? apic_timer_interrupt+0xf/0x20 [11976.233786] [11976.238484] ? native_safe_halt+0x2e/0x50 [11976.241648] kvm_wait+0xcd/0xe0 [11976.243561] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11976.248793] do_raw_spin_lock+0x177/0x190 [11976.252289] _raw_spin_lock+0x21/0x30 [11976.256979] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11976.262294] ? do_raw_spin_unlock+0x75/0x190 [11976.264589] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11976.267479] ? do_wait_intr+0xf0/0xf0 [11976.269230] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11976.271378] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11976.273583] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11976.277192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11976.279844] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11976.281741] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11976.283482] mdc_intent_lock+0x353/0x7e0 [mdc] [11976.290547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11976.293148] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11976.296491] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11976.298562] lmv_intent_open+0x338/0x1150 [lmv] [11976.299786] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11976.305146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11976.308203] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11976.310624] ll_intent_lock+0xc5/0xaa0 [lustre] [11976.312967] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11976.315689] ll_lookup_it+0xe0e/0x2270 [lustre] [11976.318560] ? do_raw_spin_unlock+0x75/0x190 [11976.320615] ? _raw_spin_unlock+0x12/0x30 [11976.322560] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11976.326923] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11976.329347] ll_atomic_open+0x36a/0xd70 [lustre] [11976.331446] lookup_open+0x356/0xb40 [11976.333388] do_last+0x313/0xfc0 [11976.335232] ? nd_jump_root+0xe5/0x160 [11976.336965] ? path_init+0x437/0x520 [11976.338388] path_openat+0xf7/0x500 [11976.339964] ? do_raw_spin_unlock+0x75/0x190 [11976.341792] do_filp_open+0x99/0x140 [11976.344232] ? getname_flags+0x6e/0x330 [11976.346044] ? __check_object_size+0xff/0x256 [11976.349672] ? do_raw_spin_unlock+0x75/0x190 [11976.355176] ? _raw_spin_unlock+0x12/0x30 [11976.357926] do_sys_openat2+0x2b4/0x410 [11976.360527] do_sys_open+0x73/0xa0 [11976.366077] __x64_sys_openat+0x24/0x30 [11976.368826] do_syscall_64+0xc1/0x440 [11976.370478] entry_SYSCALL_64_after_hwframe+0x49/0xae [11976.374254] RIP: 0033:0x7f1a63983332 [11976.376596] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11976.384983] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11976.388518] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [11976.392089] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [11976.398560] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [11976.403325] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [11976.407597] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [11979.236063] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9956s! [11979.239500] Showing busy workqueues and worker pools: [11979.241473] workqueue events: flags=0x0 [11979.244650] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11979.256337] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [11979.263009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11979.270837] pending: sec_gc_main [ptlrpc] [11979.273383] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11979.276845] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [11979.281263] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11979.283343] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [11979.287654] workqueue events_long: flags=0x0 [11979.289617] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11979.299576] pending: lu_objects_flush [obdclass] [11979.304409] workqueue events_unbound: flags=0x2 [11979.306145] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11979.310359] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11979.318991] workqueue events_power_efficient: flags=0x80 [11979.328141] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11979.340055] pending: neigh_periodic_work, do_cache_clean [11979.345530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11979.350646] pending: neigh_periodic_work, check_lifetime [11979.355395] workqueue mm_percpu_wq: flags=0x8 [11979.357140] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11979.366511] pending: vmstat_update [11979.372989] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11979.379170] pending: vmstat_update [11979.381685] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11979.388167] pending: vmstat_update [11979.390357] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11979.397326] pending: vmstat_update [11979.399130] workqueue writeback: flags=0x4a [11979.407591] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11979.416162] pending: wb_workfn, wb_workfn [11979.420037] workqueue xprtiod: flags=0xa [11979.423484] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11979.426992] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [11979.439058] workqueue ipv6_addrconf: flags=0x40008 [11979.440942] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11979.443913] pending: addrconf_verify_work [11979.449171] workqueue knbd0-recv: flags=0x1a [11979.454400] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11979.457129] in-flight: 647:recv_work [11979.458867] workqueue ptlrpc_pinger: flags=0x2 [11979.460655] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11979.463306] pending: ptlrpc_pinger_main [ptlrpc] [11979.466119] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11969s workers=2 idle: 652 [11982.669365] rcu: INFO: rcu_sched self-detected stall on CPU [11982.673000] rcu: 3-...!: (6278484 ticks this GP) idle=826/1/0x4000000000000002 softirq=131754/131754 fqs=84 [11982.682113] (t=9960276 jiffies g=269633 q=3465) [11982.688727] rcu: rcu_sched kthread starved for 9959928 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [11982.696053] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11982.703924] rcu: RCU grace-period kthread stack dump: [11982.706452] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11982.717160] Call Trace: [11982.718231] __schedule+0x351/0xcb0 [11982.722321] schedule+0xc0/0x180 [11982.726986] schedule_timeout+0xb4/0x190 [11982.730170] ? do_raw_spin_unlock+0x75/0x190 [11982.733174] ? __next_timer_interrupt+0x160/0x160 [11982.738035] ? prepare_to_swait_event+0xf3/0x140 [11982.740191] rcu_gp_kthread+0x8dd/0x15b0 [11982.743276] ? force_qs_rnp+0x2c0/0x2c0 [11982.749635] kthread+0x1d1/0x200 [11982.752761] ? set_kthread_struct+0x70/0x70 [11982.756735] ret_from_fork+0x1f/0x30 [11982.758445] rcu: Stack dump where RCU GP kthread last ran: [11982.762716] Sending NMI from CPU 3 to CPUs 0: [11982.764487] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [11982.767859] NMI backtrace for cpu 3 [11982.770050] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11982.777937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11982.781693] Call Trace: [11982.782651] [11982.787789] ? dump_stack+0xbb/0x10e [11982.792189] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11982.795367] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11982.798586] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11982.808021] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11982.811809] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11982.815334] ? rcu_dump_cpu_stacks+0x11b/0x171 [11982.820341] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11982.824153] ? update_process_times+0x6d/0xc0 [11982.827228] ? tick_sched_handle.isra.18+0x27/0x90 [11982.831030] ? tick_sched_timer+0x53/0xb0 [11982.832821] ? __hrtimer_run_queues+0x1c2/0x4d0 [11982.838407] ? tick_sched_handle.isra.18+0x90/0x90 [11982.843299] ? hrtimer_interrupt+0x128/0x2c0 [11982.847943] ? smp_apic_timer_interrupt+0xa6/0x270 [11982.855924] ? apic_timer_interrupt+0xf/0x20 [11982.861043] [11982.863214] ? native_safe_halt+0x2e/0x50 [11982.869379] ? kvm_wait+0xcd/0xe0 [11982.874867] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11982.880624] ? do_raw_spin_lock+0x177/0x190 [11982.884624] ? _raw_spin_lock+0x21/0x30 [11982.888349] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11982.901238] ? obd_mod_free+0x46/0x140 [ptlrpc] [11982.907801] ? mdc_commit_open+0x91/0xb0 [mdc] [11982.911267] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11982.918076] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11982.925104] ? after_reply+0xe2c/0x1780 [ptlrpc] [11982.929675] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11982.936659] ? __next_timer_interrupt+0x160/0x160 [11982.939968] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11982.948624] ? do_wait_intr+0xf0/0xf0 [11982.952356] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11982.958719] ? kthread+0x1d1/0x200 [11982.962107] ? set_kthread_struct+0x70/0x70 [11982.963963] ? ret_from_fork+0x1f/0x30 [11996.059643] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [11996.062276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11996.111106] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11996.117922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11996.122005] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11996.124686] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11996.136274] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11996.142238] RAX: 0000000000006012 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [11996.146331] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [11996.150780] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [11996.155873] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [11996.159938] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [11996.163994] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [11996.170724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11996.175470] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [11996.183425] Call Trace: [11996.184329] [11996.185677] ? show_regs.cold.9+0x22/0x2f [11996.187268] ? watchdog_timer_fn.cold.6+0x62/0x10a [11996.189224] ? __hrtimer_run_queues+0x1c2/0x4d0 [11996.196150] ? watchdog+0x40/0x40 [11996.197923] ? hrtimer_interrupt+0x128/0x2c0 [11996.202935] ? smp_apic_timer_interrupt+0xa6/0x270 [11996.206103] ? apic_timer_interrupt+0xf/0x20 [11996.211508] [11996.212530] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11996.219116] do_raw_spin_lock+0x177/0x190 [11996.222030] _raw_spin_lock+0x21/0x30 [11996.228229] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11996.238082] ? do_raw_spin_unlock+0x75/0x190 [11996.240020] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11996.243770] ? do_wait_intr+0xf0/0xf0 [11996.245468] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11996.248987] mdc_close+0x53b/0xd70 [mdc] [11996.252455] lmv_close+0x18d/0x490 [lmv] [11996.259922] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11996.262525] ll_md_real_close+0xa5/0x2a0 [lustre] [11996.272437] ll_md_close+0x274/0x800 [lustre] [11996.275143] ll_file_release+0x128/0xc50 [lustre] [11996.277456] __fput+0x10d/0x440 [11996.278830] ____fput+0x12/0x20 [11996.280277] task_work_run+0xc6/0x110 [11996.286742] exit_to_usermode_loop+0x1dd/0x1f0 [11996.290288] do_syscall_64+0x3d6/0x440 [11996.296122] entry_SYSCALL_64_after_hwframe+0x49/0xae [11996.300675] RIP: 0033:0x7f1a63983cd8 [11996.303067] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11996.322730] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11996.328923] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [11996.336079] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [11996.342041] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [11996.349802] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [11996.355760] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12000.059899] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37209] [12000.076075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12000.129851] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12000.140234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12000.151238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12000.159637] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12000.176377] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12000.183636] RAX: 0000000000003367 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12000.191215] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [12000.198901] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [12000.204698] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [12000.213694] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [12000.219180] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [12000.233180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12000.241904] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [12000.248154] Call Trace: [12000.248933] [12000.249464] ? show_regs.cold.9+0x22/0x2f [12000.252304] ? watchdog_timer_fn.cold.6+0x62/0x10a [12000.256261] ? __hrtimer_run_queues+0x1c2/0x4d0 [12000.264376] ? watchdog+0x40/0x40 [12000.271866] ? hrtimer_interrupt+0x128/0x2c0 [12000.278381] ? smp_apic_timer_interrupt+0xa6/0x270 [12000.282424] ? apic_timer_interrupt+0xf/0x20 [12000.287267] [12000.288412] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12000.298008] do_raw_spin_lock+0x177/0x190 [12000.301071] _raw_spin_lock+0x21/0x30 [12000.303388] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12000.310161] ? do_raw_spin_unlock+0x75/0x190 [12000.315101] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12000.319209] ? do_wait_intr+0xf0/0xf0 [12000.327149] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12000.336174] mdc_close+0x53b/0xd70 [mdc] [12000.344272] lmv_close+0x18d/0x490 [lmv] [12000.351223] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12000.364794] ll_md_real_close+0xa5/0x2a0 [lustre] [12000.373986] ll_md_close+0x274/0x800 [lustre] [12000.387226] ll_file_release+0x128/0xc50 [lustre] [12000.389120] __fput+0x10d/0x440 [12000.401940] ____fput+0x12/0x20 [12000.403190] task_work_run+0xc6/0x110 [12000.410315] exit_to_usermode_loop+0x1dd/0x1f0 [12000.413389] do_syscall_64+0x3d6/0x440 [12000.421252] entry_SYSCALL_64_after_hwframe+0x49/0xae [12000.423185] RIP: 0033:0x7f1a63983cd8 [12000.433787] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12000.451180] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12000.458909] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [12000.464986] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [12000.485594] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [12000.488141] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [12000.508130] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12008.053441] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [12008.058133] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12008.058352] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [12008.081634] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12008.081650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12008.081654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12008.081667] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12008.081671] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12008.081678] RAX: 00000000000041bd RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12008.081680] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [12008.081682] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [12008.081684] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [12008.081687] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [12008.081690] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [12008.081693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12008.081763] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [12008.081772] Call Trace: [12008.081778] [12008.081784] ? show_regs.cold.9+0x22/0x2f [12008.081794] ? watchdog_timer_fn.cold.6+0x62/0x10a [12008.081801] ? __hrtimer_run_queues+0x1c2/0x4d0 [12008.081805] ? watchdog+0x40/0x40 [12008.081809] ? hrtimer_interrupt+0x128/0x2c0 [12008.081812] ? smp_apic_timer_interrupt+0xa6/0x270 [12008.081820] ? apic_timer_interrupt+0xf/0x20 [12008.081824] [12008.081827] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12008.081834] do_raw_spin_lock+0x177/0x190 [12008.081839] _raw_spin_lock+0x21/0x30 [12008.081844] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12008.082123] ? do_raw_spin_unlock+0x75/0x190 [12008.082129] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12008.082728] ? do_wait_intr+0xf0/0xf0 [12008.082737] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12008.082939] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12008.083157] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12008.083535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12008.083648] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12008.083690] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12008.083839] mdc_intent_lock+0x353/0x7e0 [mdc] [12008.083867] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12008.083951] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12008.084179] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12008.084289] lmv_intent_open+0x338/0x1150 [lmv] [12008.084321] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12008.084407] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12008.084555] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12008.084579] ll_intent_lock+0xc5/0xaa0 [lustre] [12008.084664] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12008.084821] ll_lookup_it+0xe0e/0x2270 [lustre] [12008.084905] ? do_raw_spin_unlock+0x75/0x190 [12008.084914] ? _raw_spin_unlock+0x12/0x30 [12008.084921] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12008.084946] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12008.084964] ll_atomic_open+0x36a/0xd70 [lustre] [12008.085051] lookup_open+0x356/0xb40 [12008.085060] do_last+0x313/0xfc0 [12008.085065] ? nd_jump_root+0xe5/0x160 [12008.085070] ? path_init+0x437/0x520 [12008.085075] path_openat+0xf7/0x500 [12008.085080] ? do_raw_spin_unlock+0x75/0x190 [12008.085085] do_filp_open+0x99/0x140 [12008.085092] ? getname_flags+0x6e/0x330 [12008.085097] ? __check_object_size+0xff/0x256 [12008.085104] ? do_raw_spin_unlock+0x75/0x190 [12008.085108] ? _raw_spin_unlock+0x12/0x30 [12008.085112] do_sys_openat2+0x2b4/0x410 [12008.085116] do_sys_open+0x73/0xa0 [12008.085120] __x64_sys_openat+0x24/0x30 [12008.085124] do_syscall_64+0xc1/0x440 [12008.085131] entry_SYSCALL_64_after_hwframe+0x49/0xae [12008.085138] RIP: 0033:0x7f1a63983332 [12008.085146] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12008.085150] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12008.085155] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [12008.085158] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [12008.085160] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [12008.085163] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [12008.085165] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [12008.525350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12008.559552] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12008.570796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12008.578246] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12008.587844] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12008.615245] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12008.627622] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [12008.641091] RDX: 0000000000001765 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [12008.651784] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [12008.662982] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [12008.679611] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12008.692132] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [12008.701560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12008.718364] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [12008.722066] Call Trace: [12008.723589] [12008.728022] ? show_regs.cold.9+0x22/0x2f [12008.735143] ? watchdog_timer_fn.cold.6+0x62/0x10a [12008.741289] ? __hrtimer_run_queues+0x1c2/0x4d0 [12008.743362] ? watchdog+0x40/0x40 [12008.745314] ? hrtimer_interrupt+0x128/0x2c0 [12008.747317] ? smp_apic_timer_interrupt+0xa6/0x270 [12008.752397] ? apic_timer_interrupt+0xf/0x20 [12008.755338] [12008.760097] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12008.764019] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12008.769841] do_raw_spin_lock+0x177/0x190 [12008.771619] _raw_spin_lock+0x21/0x30 [12008.773407] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12008.777641] obd_mod_free+0x46/0x140 [ptlrpc] [12008.780824] mdc_commit_open+0x91/0xb0 [mdc] [12008.785347] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12008.791323] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12008.794678] after_reply+0xe2c/0x1780 [ptlrpc] [12008.797054] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12008.799915] ? __next_timer_interrupt+0x160/0x160 [12008.802030] ptlrpcd+0xa82/0xe40 [ptlrpc] [12008.806730] ? do_wait_intr+0xf0/0xf0 [12008.808949] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12008.812556] kthread+0x1d1/0x200 [12008.814278] ? set_kthread_struct+0x70/0x70 [12008.816652] ret_from_fork+0x1f/0x30 [12009.952043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9986s! [12009.955740] Showing busy workqueues and worker pools: [12009.958840] workqueue events: flags=0x0 [12009.961426] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12009.965317] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [12009.968996] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12009.973164] pending: sec_gc_main [ptlrpc] [12009.976535] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12009.980093] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [12009.988333] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12009.995390] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [12010.000316] workqueue events_long: flags=0x0 [12010.002334] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12010.005836] pending: lu_objects_flush [obdclass] [12010.009062] workqueue events_unbound: flags=0x2 [12010.011331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12010.016190] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12010.023153] workqueue events_power_efficient: flags=0x80 [12010.026649] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12010.029959] pending: neigh_periodic_work, do_cache_clean [12010.033135] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12010.037409] pending: neigh_periodic_work, check_lifetime [12010.041699] workqueue mm_percpu_wq: flags=0x8 [12010.044274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12010.047858] pending: vmstat_update [12010.049906] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12010.053123] pending: vmstat_update [12010.055502] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12010.058859] pending: vmstat_update [12010.060619] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12010.064320] pending: vmstat_update [12010.066332] workqueue writeback: flags=0x4a [12010.068522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12010.071313] pending: wb_workfn, wb_workfn [12010.073964] workqueue xprtiod: flags=0xa [12010.075958] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12010.079126] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12010.083368] workqueue ipv6_addrconf: flags=0x40008 [12010.085640] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12010.089167] pending: addrconf_verify_work [12010.091528] workqueue knbd0-recv: flags=0x1a [12010.094097] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12010.097590] in-flight: 647:recv_work [12010.099620] workqueue ptlrpc_pinger: flags=0x2 [12010.101631] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12010.104886] pending: ptlrpc_pinger_main [ptlrpc] [12010.107845] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12000s workers=2 idle: 652 [12024.057686] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [writemany:37208] [12024.068698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12024.139806] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12024.163016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12024.178194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12024.187016] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12024.214871] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12024.219763] RAX: 0000000000006513 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12024.224706] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [12024.230482] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [12024.236720] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [12024.241067] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [12024.245598] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [12024.252866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12024.263790] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [12024.277792] Call Trace: [12024.282830] [12024.288186] ? show_regs.cold.9+0x22/0x2f [12024.292452] ? watchdog_timer_fn.cold.6+0x62/0x10a [12024.295648] ? __hrtimer_run_queues+0x1c2/0x4d0 [12024.300718] ? watchdog+0x40/0x40 [12024.307471] ? hrtimer_interrupt+0x128/0x2c0 [12024.312945] ? smp_apic_timer_interrupt+0xa6/0x270 [12024.321910] ? apic_timer_interrupt+0xf/0x20 [12024.327027] [12024.328765] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12024.334461] do_raw_spin_lock+0x177/0x190 [12024.341933] _raw_spin_lock+0x21/0x30 [12024.347181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12024.350993] ? do_raw_spin_unlock+0x75/0x190 [12024.353725] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12024.359642] ? do_wait_intr+0xf0/0xf0 [12024.362228] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12024.368140] mdc_close+0x53b/0xd70 [mdc] [12024.377573] lmv_close+0x18d/0x490 [lmv] [12024.379162] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12024.394330] ll_md_real_close+0xa5/0x2a0 [lustre] [12024.396111] ll_md_close+0x274/0x800 [lustre] [12024.407938] ll_file_release+0x128/0xc50 [lustre] [12024.415826] __fput+0x10d/0x440 [12024.418763] ____fput+0x12/0x20 [12024.422593] task_work_run+0xc6/0x110 [12024.425694] exit_to_usermode_loop+0x1dd/0x1f0 [12024.429443] do_syscall_64+0x3d6/0x440 [12024.433025] entry_SYSCALL_64_after_hwframe+0x49/0xae [12024.441900] RIP: 0033:0x7f1a63983cd8 [12024.447310] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12024.477169] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12024.495450] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [12024.509711] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [12024.517736] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [12024.530100] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [12024.538420] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12028.056013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [12028.064853] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12028.108832] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12028.114604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12028.123875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12028.127389] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12028.138603] RSP: 0018:ffffc38542073af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12028.142127] RAX: 0000000000000385 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12028.144885] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [12028.148843] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [12028.154145] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [12028.160225] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [12028.165586] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [12028.169653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12028.172482] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [12028.175944] Call Trace: [12028.177008] [12028.178125] ? show_regs.cold.9+0x22/0x2f [12028.179890] ? watchdog_timer_fn.cold.6+0x62/0x10a [12028.181416] ? __hrtimer_run_queues+0x1c2/0x4d0 [12028.183622] ? watchdog+0x40/0x40 [12028.184757] ? hrtimer_interrupt+0x128/0x2c0 [12028.186860] ? smp_apic_timer_interrupt+0xa6/0x270 [12028.189224] ? apic_timer_interrupt+0xf/0x20 [12028.191443] [12028.192500] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12028.194471] do_raw_spin_lock+0x177/0x190 [12028.195736] _raw_spin_lock+0x21/0x30 [12028.197362] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12028.199970] ? do_raw_spin_unlock+0x75/0x190 [12028.202532] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12028.204982] ? do_wait_intr+0xf0/0xf0 [12028.206376] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12028.209652] mdc_close+0x53b/0xd70 [mdc] [12028.214721] lmv_close+0x18d/0x490 [lmv] [12028.218800] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12028.221625] ll_md_real_close+0xa5/0x2a0 [lustre] [12028.228402] ll_md_close+0x274/0x800 [lustre] [12028.230508] ll_file_release+0x128/0xc50 [lustre] [12028.233951] __fput+0x10d/0x440 [12028.235354] ____fput+0x12/0x20 [12028.236462] task_work_run+0xc6/0x110 [12028.237702] exit_to_usermode_loop+0x1dd/0x1f0 [12028.239460] do_syscall_64+0x3d6/0x440 [12028.241350] entry_SYSCALL_64_after_hwframe+0x49/0xae [12028.243182] RIP: 0033:0x7f1a63983cd8 [12028.244711] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12028.252512] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12028.255637] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [12028.258957] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [12028.262181] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [12028.265766] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [12028.269218] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12036.053025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [12036.055797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12036.058165] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [12036.089819] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12036.089838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12036.089840] RIP: 0010:native_safe_halt+0x2e/0x50 [12036.089854] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12036.089859] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12036.089865] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12036.089871] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [12036.089874] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [12036.089876] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [12036.089879] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [12036.089882] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [12036.089885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12036.089887] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [12036.089896] Call Trace: [12036.089902] [12036.089912] ? show_regs.cold.9+0x22/0x2f [12036.089925] ? watchdog_timer_fn.cold.6+0x62/0x10a [12036.089931] ? __hrtimer_run_queues+0x1c2/0x4d0 [12036.089936] ? watchdog+0x40/0x40 [12036.089940] ? hrtimer_interrupt+0x128/0x2c0 [12036.089944] ? smp_apic_timer_interrupt+0xa6/0x270 [12036.089958] ? apic_timer_interrupt+0xf/0x20 [12036.089963] [12036.089966] ? native_safe_halt+0x2e/0x50 [12036.089970] kvm_wait+0xcd/0xe0 [12036.089980] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12036.089991] do_raw_spin_lock+0x177/0x190 [12036.089996] _raw_spin_lock+0x21/0x30 [12036.090001] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12036.095556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12036.095637] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12036.095645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12036.095648] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12036.095669] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12036.095674] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12036.095678] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [12036.095681] RDX: 0000000000003208 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [12036.095683] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [12036.095685] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [12036.095687] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12036.095690] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [12036.095693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12036.095695] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [12036.095707] Call Trace: [12036.095712] [12036.095718] ? show_regs.cold.9+0x22/0x2f [12036.095728] ? watchdog_timer_fn.cold.6+0x62/0x10a [12036.095734] ? __hrtimer_run_queues+0x1c2/0x4d0 [12036.095738] ? watchdog+0x40/0x40 [12036.095743] ? hrtimer_interrupt+0x128/0x2c0 [12036.095752] ? smp_apic_timer_interrupt+0xa6/0x270 [12036.095760] ? apic_timer_interrupt+0xf/0x20 [12036.095765] [12036.095769] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12036.095774] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12036.095779] do_raw_spin_lock+0x177/0x190 [12036.095787] _raw_spin_lock+0x21/0x30 [12036.095792] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12036.096130] obd_mod_free+0x46/0x140 [ptlrpc] [12036.096415] mdc_commit_open+0x91/0xb0 [mdc] [12036.096455] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12036.096648] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12036.096858] after_reply+0xe2c/0x1780 [ptlrpc] [12036.097061] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12036.097360] ? __next_timer_interrupt+0x160/0x160 [12036.097374] ptlrpcd+0xa82/0xe40 [ptlrpc] [12036.099078] ? do_wait_intr+0xf0/0xf0 [12036.099090] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12036.099812] kthread+0x1d1/0x200 [12036.099884] ? set_kthread_struct+0x70/0x70 [12036.099894] ret_from_fork+0x1f/0x30 [12036.318889] ? do_raw_spin_unlock+0x75/0x190 [12036.320623] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12036.323357] ? do_wait_intr+0xf0/0xf0 [12036.325242] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12036.328144] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12036.331217] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12036.334677] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12036.336912] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12036.339626] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12036.341822] mdc_intent_lock+0x353/0x7e0 [mdc] [12036.343874] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12036.346547] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12036.350353] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12036.352898] lmv_intent_open+0x338/0x1150 [lmv] [12036.355471] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12036.358242] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12036.360902] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12036.363647] ll_intent_lock+0xc5/0xaa0 [lustre] [12036.365947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12036.368318] ll_lookup_it+0xe0e/0x2270 [lustre] [12036.370710] ? do_raw_spin_unlock+0x75/0x190 [12036.372770] ? _raw_spin_unlock+0x12/0x30 [12036.374542] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12036.377079] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12036.379316] ll_atomic_open+0x36a/0xd70 [lustre] [12036.381631] lookup_open+0x356/0xb40 [12036.383769] do_last+0x313/0xfc0 [12036.385544] ? nd_jump_root+0xe5/0x160 [12036.387305] ? path_init+0x437/0x520 [12036.388847] path_openat+0xf7/0x500 [12036.391037] ? do_raw_spin_unlock+0x75/0x190 [12036.393054] do_filp_open+0x99/0x140 [12036.394796] ? getname_flags+0x6e/0x330 [12036.396765] ? __check_object_size+0xff/0x256 [12036.398923] ? do_raw_spin_unlock+0x75/0x190 [12036.401481] ? _raw_spin_unlock+0x12/0x30 [12036.403271] do_sys_openat2+0x2b4/0x410 [12036.405618] do_sys_open+0x73/0xa0 [12036.407260] __x64_sys_openat+0x24/0x30 [12036.409008] do_syscall_64+0xc1/0x440 [12036.410948] entry_SYSCALL_64_after_hwframe+0x49/0xae [12036.413653] RIP: 0033:0x7f1a63983332 [12036.415046] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12036.422778] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12036.425791] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [12036.428625] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [12036.431661] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [12036.434869] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [12036.438112] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [12040.162536] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10017s! [12040.171090] Showing busy workqueues and worker pools: [12040.174779] workqueue events: flags=0x0 [12040.180540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12040.184023] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [12040.195423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12040.201839] pending: sec_gc_main [ptlrpc] [12040.208454] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12040.216717] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [12040.225332] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12040.231481] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [12040.238570] workqueue events_long: flags=0x0 [12040.240772] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12040.246646] pending: lu_objects_flush [obdclass] [12040.250829] workqueue events_unbound: flags=0x2 [12040.252471] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12040.259298] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12040.265320] workqueue events_power_efficient: flags=0x80 [12040.268972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12040.283485] pending: neigh_periodic_work, do_cache_clean [12040.289316] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12040.295330] pending: neigh_periodic_work, check_lifetime [12040.298500] workqueue mm_percpu_wq: flags=0x8 [12040.300686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12040.304620] pending: vmstat_update [12040.306315] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12040.309459] pending: vmstat_update [12040.311244] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12040.314290] pending: vmstat_update [12040.315744] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12040.318366] pending: vmstat_update [12040.319736] workqueue writeback: flags=0x4a [12040.321328] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12040.324111] pending: wb_workfn, wb_workfn [12040.325925] workqueue xprtiod: flags=0xa [12040.327518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12040.329939] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12040.338063] workqueue ipv6_addrconf: flags=0x40008 [12040.341431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12040.348875] pending: addrconf_verify_work [12040.351499] workqueue knbd0-recv: flags=0x1a [12040.353774] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12040.359592] in-flight: 647:recv_work [12040.361611] workqueue ptlrpc_pinger: flags=0x2 [12040.363643] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12040.368008] pending: ptlrpc_pinger_main [ptlrpc] [12040.370921] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12030s workers=2 idle: 652 [12052.057155] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [12052.064755] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12052.105659] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12052.114725] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12052.122689] RIP: 0010:native_safe_halt+0x2e/0x50 [12052.124775] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12052.151684] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12052.168757] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12052.172877] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [12052.177873] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [12052.185351] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [12052.190687] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [12052.199669] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [12052.211651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12052.216926] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [12052.223529] Call Trace: [12052.224566] [12052.225580] ? show_regs.cold.9+0x22/0x2f [12052.229146] ? watchdog_timer_fn.cold.6+0x62/0x10a [12052.232878] ? __hrtimer_run_queues+0x1c2/0x4d0 [12052.240740] ? watchdog+0x40/0x40 [12052.242739] ? hrtimer_interrupt+0x128/0x2c0 [12052.247726] ? smp_apic_timer_interrupt+0xa6/0x270 [12052.251548] ? apic_timer_interrupt+0xf/0x20 [12052.257276] [12052.259159] ? native_safe_halt+0x2e/0x50 [12052.260629] kvm_wait+0xcd/0xe0 [12052.263692] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12052.266965] do_raw_spin_lock+0x177/0x190 [12052.269356] _raw_spin_lock+0x21/0x30 [12052.271857] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12052.279976] ? do_raw_spin_unlock+0x75/0x190 [12052.282259] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12052.289251] ? do_wait_intr+0xf0/0xf0 [12052.292996] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12052.302115] mdc_close+0x53b/0xd70 [mdc] [12052.311518] lmv_close+0x18d/0x490 [lmv] [12052.313018] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12052.323533] ll_md_real_close+0xa5/0x2a0 [lustre] [12052.332002] ll_md_close+0x274/0x800 [lustre] [12052.342766] ll_file_release+0x128/0xc50 [lustre] [12052.348831] __fput+0x10d/0x440 [12052.354517] ____fput+0x12/0x20 [12052.355752] task_work_run+0xc6/0x110 [12052.365548] exit_to_usermode_loop+0x1dd/0x1f0 [12052.370422] do_syscall_64+0x3d6/0x440 [12052.371820] entry_SYSCALL_64_after_hwframe+0x49/0xae [12052.381800] RIP: 0033:0x7f1a63983cd8 [12052.382773] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12052.403634] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12052.409914] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [12052.416772] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [12052.423434] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [12052.429463] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [12052.433833] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12056.057148] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [12056.064800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12056.097906] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12056.104843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12056.108467] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12056.110478] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12056.122358] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12056.129423] RAX: 0000000000002fb6 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12056.133356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [12056.135341] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [12056.139309] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [12056.141870] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [12056.143591] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [12056.147534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12056.151458] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [12056.156216] Call Trace: [12056.157403] [12056.160530] ? show_regs.cold.9+0x22/0x2f [12056.162522] ? watchdog_timer_fn.cold.6+0x62/0x10a [12056.165730] ? __hrtimer_run_queues+0x1c2/0x4d0 [12056.167835] ? watchdog+0x40/0x40 [12056.169434] ? hrtimer_interrupt+0x128/0x2c0 [12056.173372] ? smp_apic_timer_interrupt+0xa6/0x270 [12056.177436] ? apic_timer_interrupt+0xf/0x20 [12056.181862] [12056.187281] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12056.190561] do_raw_spin_lock+0x177/0x190 [12056.192662] _raw_spin_lock+0x21/0x30 [12056.194393] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12056.196710] ? do_raw_spin_unlock+0x75/0x190 [12056.198679] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12056.201059] ? do_wait_intr+0xf0/0xf0 [12056.202208] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12056.204359] mdc_close+0x53b/0xd70 [mdc] [12056.205764] lmv_close+0x18d/0x490 [lmv] [12056.207427] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12056.209848] ll_md_real_close+0xa5/0x2a0 [lustre] [12056.213571] ll_md_close+0x274/0x800 [lustre] [12056.215638] ll_file_release+0x128/0xc50 [lustre] [12056.217967] __fput+0x10d/0x440 [12056.219414] ____fput+0x12/0x20 [12056.220567] task_work_run+0xc6/0x110 [12056.222391] exit_to_usermode_loop+0x1dd/0x1f0 [12056.224413] do_syscall_64+0x3d6/0x440 [12056.225690] entry_SYSCALL_64_after_hwframe+0x49/0xae [12056.227867] RIP: 0033:0x7f1a63983cd8 [12056.229118] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12056.242339] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12056.248605] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [12056.252849] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [12056.256298] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [12056.259554] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [12056.261500] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12064.058021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [12064.062348] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12064.097274] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12064.103630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12064.112528] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12064.115406] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12064.128456] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12064.133633] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [12064.139885] RDX: 0000000000005fd9 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [12064.145216] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [12064.148710] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [12064.157659] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12064.165592] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [12064.168977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12064.171640] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [12064.177684] Call Trace: [12064.184839] [12064.186414] ? show_regs.cold.9+0x22/0x2f [12064.187684] ? watchdog_timer_fn.cold.6+0x62/0x10a [12064.189797] ? __hrtimer_run_queues+0x1c2/0x4d0 [12064.193996] ? watchdog+0x40/0x40 [12064.198499] ? hrtimer_interrupt+0x128/0x2c0 [12064.200053] ? smp_apic_timer_interrupt+0xa6/0x270 [12064.202816] ? apic_timer_interrupt+0xf/0x20 [12064.204807] [12064.207577] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12064.211948] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12064.215005] do_raw_spin_lock+0x177/0x190 [12064.216894] _raw_spin_lock+0x21/0x30 [12064.218678] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12064.221605] obd_mod_free+0x46/0x140 [ptlrpc] [12064.224619] mdc_commit_open+0x91/0xb0 [mdc] [12064.226249] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12064.228843] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12064.231701] after_reply+0xe2c/0x1780 [ptlrpc] [12064.233710] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12064.241538] ? __next_timer_interrupt+0x160/0x160 [12064.246478] ptlrpcd+0xa82/0xe40 [ptlrpc] [12064.249208] ? do_wait_intr+0xf0/0xf0 [12064.254158] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12064.260511] kthread+0x1d1/0x200 [12064.262486] ? set_kthread_struct+0x70/0x70 [12064.271949] ret_from_fork+0x1f/0x30 [12068.053029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [12068.059823] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12068.101280] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12068.115221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12068.128642] RIP: 0010:native_safe_halt+0x2e/0x50 [12068.135058] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12068.149132] RSP: 0018:ffffc38541c374f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12068.159219] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12068.162415] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482034154 [12068.168367] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [12068.173343] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [12068.182995] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [12068.193953] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [12068.200210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12068.203614] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [12068.208241] Call Trace: [12068.212664] [12068.214410] ? show_regs.cold.9+0x22/0x2f [12068.220293] ? watchdog_timer_fn.cold.6+0x62/0x10a [12068.223692] ? __hrtimer_run_queues+0x1c2/0x4d0 [12068.225852] ? watchdog+0x40/0x40 [12068.231005] ? hrtimer_interrupt+0x128/0x2c0 [12068.234049] ? smp_apic_timer_interrupt+0xa6/0x270 [12068.237417] ? apic_timer_interrupt+0xf/0x20 [12068.240065] [12068.241047] ? native_safe_halt+0x2e/0x50 [12068.245114] kvm_wait+0xcd/0xe0 [12068.248769] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12068.253431] do_raw_spin_lock+0x177/0x190 [12068.255939] _raw_spin_lock+0x21/0x30 [12068.257624] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12068.266698] ? do_raw_spin_unlock+0x75/0x190 [12068.271286] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12068.277231] ? do_wait_intr+0xf0/0xf0 [12068.285558] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12068.291447] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12068.297046] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12068.301781] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12068.306596] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12068.313623] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12068.318241] mdc_intent_lock+0x353/0x7e0 [mdc] [12068.324710] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12068.330140] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12068.340050] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12068.345182] lmv_intent_open+0x338/0x1150 [lmv] [12068.352268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12068.357607] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12068.362716] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12068.365287] ll_intent_lock+0xc5/0xaa0 [lustre] [12068.370726] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12068.373706] ll_lookup_it+0xe0e/0x2270 [lustre] [12068.375949] ? do_raw_spin_unlock+0x75/0x190 [12068.380839] ? _raw_spin_unlock+0x12/0x30 [12068.385427] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12068.389022] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12068.395093] ll_atomic_open+0x36a/0xd70 [lustre] [12068.397992] lookup_open+0x356/0xb40 [12068.401999] do_last+0x313/0xfc0 [12068.405578] ? nd_jump_root+0xe5/0x160 [12068.407085] ? path_init+0x437/0x520 [12068.411380] path_openat+0xf7/0x500 [12068.413252] ? do_raw_spin_unlock+0x75/0x190 [12068.417841] do_filp_open+0x99/0x140 [12068.419740] ? getname_flags+0x6e/0x330 [12068.423454] ? __check_object_size+0xff/0x256 [12068.426740] ? do_raw_spin_unlock+0x75/0x190 [12068.428441] ? _raw_spin_unlock+0x12/0x30 [12068.430985] do_sys_openat2+0x2b4/0x410 [12068.432787] do_sys_open+0x73/0xa0 [12068.437163] __x64_sys_openat+0x24/0x30 [12068.439205] do_syscall_64+0xc1/0x440 [12068.442046] entry_SYSCALL_64_after_hwframe+0x49/0xae [12068.447356] RIP: 0033:0x7f1a63983332 [12068.449328] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12068.457507] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12068.461404] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [12068.464945] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [12068.469597] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [12068.473565] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [12068.478954] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [12070.881048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10047s! [12070.883593] Showing busy workqueues and worker pools: [12070.889740] workqueue events: flags=0x0 [12070.891267] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12070.898269] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [12070.907599] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12070.911540] pending: sec_gc_main [ptlrpc] [12070.915213] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12070.922209] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [12070.930286] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12070.935728] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [12070.943436] workqueue events_long: flags=0x0 [12070.946112] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12070.954535] pending: lu_objects_flush [obdclass] [12070.958153] workqueue events_unbound: flags=0x2 [12070.960644] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12070.965818] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12070.976270] workqueue events_power_efficient: flags=0x80 [12070.982745] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12070.992037] pending: neigh_periodic_work, do_cache_clean [12070.994154] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12071.012782] pending: neigh_periodic_work, check_lifetime [12071.022228] workqueue mm_percpu_wq: flags=0x8 [12071.025817] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12071.031078] pending: vmstat_update [12071.037584] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12071.042792] pending: vmstat_update [12071.045697] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12071.050965] pending: vmstat_update [12071.060663] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12071.067376] pending: vmstat_update [12071.071634] workqueue writeback: flags=0x4a [12071.078328] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12071.086370] pending: wb_workfn, wb_workfn [12071.088040] workqueue xprtiod: flags=0xa [12071.093184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12071.099132] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12071.106828] workqueue ipv6_addrconf: flags=0x40008 [12071.111345] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12071.117687] pending: addrconf_verify_work [12071.120851] workqueue knbd0-recv: flags=0x1a [12071.126733] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12071.132429] in-flight: 647:recv_work [12071.134265] workqueue ptlrpc_pinger: flags=0x2 [12071.140732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12071.143460] pending: ptlrpc_pinger_main [ptlrpc] [12071.149395] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12061s workers=2 idle: 652 [12080.057237] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [12080.060089] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12080.113697] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12080.119077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12080.122711] RIP: 0010:native_safe_halt+0x2e/0x50 [12080.124841] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12080.133728] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12080.139629] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12080.155394] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [12080.167856] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [12080.173637] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [12080.176740] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [12080.180848] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [12080.191694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12080.198580] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [12080.206174] Call Trace: [12080.210765] [12080.211703] ? show_regs.cold.9+0x22/0x2f [12080.220410] ? watchdog_timer_fn.cold.6+0x62/0x10a [12080.225477] ? __hrtimer_run_queues+0x1c2/0x4d0 [12080.228698] ? watchdog+0x40/0x40 [12080.231222] ? hrtimer_interrupt+0x128/0x2c0 [12080.234618] ? smp_apic_timer_interrupt+0xa6/0x270 [12080.241341] ? apic_timer_interrupt+0xf/0x20 [12080.247366] [12080.248069] ? native_safe_halt+0x2e/0x50 [12080.252211] kvm_wait+0xcd/0xe0 [12080.255652] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12080.263704] do_raw_spin_lock+0x177/0x190 [12080.265273] _raw_spin_lock+0x21/0x30 [12080.274048] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12080.279815] ? do_raw_spin_unlock+0x75/0x190 [12080.281139] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12080.289547] ? do_wait_intr+0xf0/0xf0 [12080.292662] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12080.301217] mdc_close+0x53b/0xd70 [mdc] [12080.315203] lmv_close+0x18d/0x490 [lmv] [12080.319927] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12080.326602] ll_md_real_close+0xa5/0x2a0 [lustre] [12080.328705] ll_md_close+0x274/0x800 [lustre] [12080.351824] ll_file_release+0x128/0xc50 [lustre] [12080.356812] __fput+0x10d/0x440 [12080.366305] ____fput+0x12/0x20 [12080.370975] task_work_run+0xc6/0x110 [12080.372396] exit_to_usermode_loop+0x1dd/0x1f0 [12080.382959] do_syscall_64+0x3d6/0x440 [12080.393056] entry_SYSCALL_64_after_hwframe+0x49/0xae [12080.396822] RIP: 0033:0x7f1a63983cd8 [12080.401133] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12080.424588] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12080.434885] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [12080.439388] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [12080.449570] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [12080.458687] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [12080.466170] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12084.056045] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [12084.062638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12084.093200] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12084.098207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12084.102163] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12084.104945] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12084.114209] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12084.117580] RAX: 0000000000000317 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12084.120862] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [12084.124206] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [12084.127699] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [12084.130908] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [12084.134387] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [12084.138226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12084.141175] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [12084.145156] Call Trace: [12084.146423] [12084.147321] ? show_regs.cold.9+0x22/0x2f [12084.149421] ? watchdog_timer_fn.cold.6+0x62/0x10a [12084.152114] ? __hrtimer_run_queues+0x1c2/0x4d0 [12084.154760] ? watchdog+0x40/0x40 [12084.156483] ? hrtimer_interrupt+0x128/0x2c0 [12084.158597] ? smp_apic_timer_interrupt+0xa6/0x270 [12084.160885] ? apic_timer_interrupt+0xf/0x20 [12084.163095] [12084.164116] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12084.167506] do_raw_spin_lock+0x177/0x190 [12084.169144] _raw_spin_lock+0x21/0x30 [12084.170881] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12084.173720] ? do_raw_spin_unlock+0x75/0x190 [12084.175924] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12084.178696] ? do_wait_intr+0xf0/0xf0 [12084.180597] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12084.186136] mdc_close+0x53b/0xd70 [mdc] [12084.188330] lmv_close+0x18d/0x490 [lmv] [12084.189917] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12084.193032] ll_md_real_close+0xa5/0x2a0 [lustre] [12084.196695] ll_md_close+0x274/0x800 [lustre] [12084.200569] ll_file_release+0x128/0xc50 [lustre] [12084.204147] __fput+0x10d/0x440 [12084.206293] ____fput+0x12/0x20 [12084.210115] task_work_run+0xc6/0x110 [12084.213926] exit_to_usermode_loop+0x1dd/0x1f0 [12084.216337] do_syscall_64+0x3d6/0x440 [12084.219504] entry_SYSCALL_64_after_hwframe+0x49/0xae [12084.224615] RIP: 0033:0x7f1a63983cd8 [12084.228951] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12084.243698] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12084.247703] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [12084.252814] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [12084.257555] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [12084.263589] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [12084.268061] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12092.058655] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [12092.062694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12092.087922] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12092.093394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12092.098440] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12092.101519] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12092.111831] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12092.118130] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [12092.123105] RDX: 0000000000006534 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [12092.133158] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [12092.137476] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [12092.144877] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12092.150089] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [12092.153787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12092.156761] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [12092.161622] Call Trace: [12092.162715] [12092.163906] ? show_regs.cold.9+0x22/0x2f [12092.165823] ? watchdog_timer_fn.cold.6+0x62/0x10a [12092.168706] ? __hrtimer_run_queues+0x1c2/0x4d0 [12092.172790] ? watchdog+0x40/0x40 [12092.174877] ? hrtimer_interrupt+0x128/0x2c0 [12092.178972] ? smp_apic_timer_interrupt+0xa6/0x270 [12092.181995] ? apic_timer_interrupt+0xf/0x20 [12092.186499] [12092.187898] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12092.194069] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12092.201997] do_raw_spin_lock+0x177/0x190 [12092.204872] _raw_spin_lock+0x21/0x30 [12092.206613] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12092.209052] obd_mod_free+0x46/0x140 [ptlrpc] [12092.211804] mdc_commit_open+0x91/0xb0 [mdc] [12092.213880] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12092.216069] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12092.221965] after_reply+0xe2c/0x1780 [ptlrpc] [12092.224480] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12092.227526] ? __next_timer_interrupt+0x160/0x160 [12092.232460] ptlrpcd+0xa82/0xe40 [ptlrpc] [12092.234673] ? do_wait_intr+0xf0/0xf0 [12092.236912] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12092.240541] kthread+0x1d1/0x200 [12092.244066] ? set_kthread_struct+0x70/0x70 [12092.249786] ret_from_fork+0x1f/0x30 [12096.054874] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [12096.070561] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12096.117530] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12096.133989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12096.143894] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12096.152515] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12096.172540] RSP: 0018:ffffc38541c37500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12096.180306] RAX: 0000000000005c12 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12096.193868] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [12096.201166] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [12096.210983] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [12096.213429] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [12096.218466] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [12096.228353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12096.232881] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [12096.238497] Call Trace: [12096.240967] [12096.241969] ? show_regs.cold.9+0x22/0x2f [12096.244341] ? watchdog_timer_fn.cold.6+0x62/0x10a [12096.246517] ? __hrtimer_run_queues+0x1c2/0x4d0 [12096.248995] ? watchdog+0x40/0x40 [12096.251498] ? hrtimer_interrupt+0x128/0x2c0 [12096.254511] ? smp_apic_timer_interrupt+0xa6/0x270 [12096.258213] ? apic_timer_interrupt+0xf/0x20 [12096.264210] [12096.265783] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12096.270530] do_raw_spin_lock+0x177/0x190 [12096.280397] _raw_spin_lock+0x21/0x30 [12096.285808] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12096.288575] ? do_raw_spin_unlock+0x75/0x190 [12096.294023] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12096.302992] ? do_wait_intr+0xf0/0xf0 [12096.308993] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12096.317363] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12096.327601] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12096.338253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12096.340166] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12096.349431] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12096.357175] mdc_intent_lock+0x353/0x7e0 [mdc] [12096.362717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12096.368434] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12096.372337] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12096.385061] lmv_intent_open+0x338/0x1150 [lmv] [12096.388439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12096.394159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12096.400011] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12096.402432] ll_intent_lock+0xc5/0xaa0 [lustre] [12096.408373] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12096.411259] ll_lookup_it+0xe0e/0x2270 [lustre] [12096.415243] ? do_raw_spin_unlock+0x75/0x190 [12096.419568] ? _raw_spin_unlock+0x12/0x30 [12096.422002] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12096.425416] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12096.429576] ll_atomic_open+0x36a/0xd70 [lustre] [12096.431460] lookup_open+0x356/0xb40 [12096.437475] do_last+0x313/0xfc0 [12096.439285] ? nd_jump_root+0xe5/0x160 [12096.441279] ? path_init+0x437/0x520 [12096.442760] path_openat+0xf7/0x500 [12096.444778] ? do_raw_spin_unlock+0x75/0x190 [12096.448419] do_filp_open+0x99/0x140 [12096.449361] ? getname_flags+0x6e/0x330 [12096.454419] ? __check_object_size+0xff/0x256 [12096.457330] ? do_raw_spin_unlock+0x75/0x190 [12096.459601] ? _raw_spin_unlock+0x12/0x30 [12096.462184] do_sys_openat2+0x2b4/0x410 [12096.464975] do_sys_open+0x73/0xa0 [12096.468010] __x64_sys_openat+0x24/0x30 [12096.469388] do_syscall_64+0xc1/0x440 [12096.472428] entry_SYSCALL_64_after_hwframe+0x49/0xae [12096.475891] RIP: 0033:0x7f1a63983332 [12096.481270] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12096.493448] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12096.512944] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [12096.520336] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [12096.525255] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [12096.531517] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [12096.540531] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [12101.607237] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10078s! [12101.616095] Showing busy workqueues and worker pools: [12101.620312] workqueue events: flags=0x0 [12101.626020] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12101.631253] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [12101.640252] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12101.644485] pending: sec_gc_main [ptlrpc] [12101.648265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12101.656202] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [12101.665331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12101.669741] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [12101.678227] workqueue events_long: flags=0x0 [12101.680756] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12101.689212] pending: lu_objects_flush [obdclass] [12101.697261] workqueue events_unbound: flags=0x2 [12101.700656] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12101.704343] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12101.712096] workqueue events_power_efficient: flags=0x80 [12101.718843] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12101.724476] pending: neigh_periodic_work, do_cache_clean [12101.728141] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12101.736116] pending: neigh_periodic_work, check_lifetime [12101.738457] workqueue mm_percpu_wq: flags=0x8 [12101.740336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12101.746015] pending: vmstat_update [12101.748762] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12101.753077] pending: vmstat_update [12101.755601] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12101.759140] pending: vmstat_update [12101.761467] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12101.768006] pending: vmstat_update [12101.769471] workqueue writeback: flags=0x4a [12101.774466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12101.780238] pending: wb_workfn, wb_workfn [12101.783709] workqueue xprtiod: flags=0xa [12101.787316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12101.792498] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12101.799939] workqueue ipv6_addrconf: flags=0x40008 [12101.809604] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12101.813236] pending: addrconf_verify_work [12101.815132] workqueue knbd0-recv: flags=0x1a [12101.816789] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12101.821014] in-flight: 647:recv_work [12101.823739] workqueue ptlrpc_pinger: flags=0x2 [12101.834462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12101.845235] pending: ptlrpc_pinger_main [ptlrpc] [12101.853148] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12091s workers=2 idle: 652 [12108.061430] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [12108.072578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12108.119467] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12108.135221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12108.144580] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12108.151138] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12108.168714] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12108.178100] RAX: 0000000000007834 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12108.183798] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [12108.187769] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [12108.196322] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [12108.203745] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [12108.211141] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [12108.222295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12108.227798] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [12108.234720] Call Trace: [12108.235988] [12108.237373] ? show_regs.cold.9+0x22/0x2f [12108.242657] ? watchdog_timer_fn.cold.6+0x62/0x10a [12108.245055] ? __hrtimer_run_queues+0x1c2/0x4d0 [12108.252453] ? watchdog+0x40/0x40 [12108.253803] ? hrtimer_interrupt+0x128/0x2c0 [12108.267378] ? smp_apic_timer_interrupt+0xa6/0x270 [12108.274597] ? apic_timer_interrupt+0xf/0x20 [12108.279416] [12108.280106] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12108.297456] do_raw_spin_lock+0x177/0x190 [12108.299009] _raw_spin_lock+0x21/0x30 [12108.300763] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12108.303317] ? do_raw_spin_unlock+0x75/0x190 [12108.304831] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12108.306650] ? do_wait_intr+0xf0/0xf0 [12108.308233] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12108.310319] mdc_close+0x53b/0xd70 [mdc] [12108.312850] lmv_close+0x18d/0x490 [lmv] [12108.317292] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12108.319726] ll_md_real_close+0xa5/0x2a0 [lustre] [12108.322158] ll_md_close+0x274/0x800 [lustre] [12108.325831] ll_file_release+0x128/0xc50 [lustre] [12108.327712] __fput+0x10d/0x440 [12108.328950] ____fput+0x12/0x20 [12108.330954] task_work_run+0xc6/0x110 [12108.332718] exit_to_usermode_loop+0x1dd/0x1f0 [12108.334790] do_syscall_64+0x3d6/0x440 [12108.336614] entry_SYSCALL_64_after_hwframe+0x49/0xae [12108.338774] RIP: 0033:0x7f1a63983cd8 [12108.340843] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12108.350093] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12108.353273] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [12108.359846] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [12108.367068] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [12108.373344] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [12108.381326] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12112.056254] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [12112.065424] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12112.130749] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12112.140826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12112.145695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12112.149041] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12112.157743] RSP: 0018:ffffc38542073af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12112.160841] RAX: 0000000000003262 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12112.163375] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [12112.165969] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [12112.167961] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [12112.169924] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [12112.171939] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [12112.174147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12112.175831] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [12112.178267] Call Trace: [12112.178982] [12112.179589] ? show_regs.cold.9+0x22/0x2f [12112.180670] ? watchdog_timer_fn.cold.6+0x62/0x10a [12112.182267] ? __hrtimer_run_queues+0x1c2/0x4d0 [12112.183397] ? watchdog+0x40/0x40 [12112.184216] ? hrtimer_interrupt+0x128/0x2c0 [12112.186139] ? smp_apic_timer_interrupt+0xa6/0x270 [12112.187756] ? apic_timer_interrupt+0xf/0x20 [12112.189219] [12112.190161] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12112.191944] do_raw_spin_lock+0x177/0x190 [12112.193264] _raw_spin_lock+0x21/0x30 [12112.194565] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12112.196606] ? do_raw_spin_unlock+0x75/0x190 [12112.198245] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12112.200633] ? do_wait_intr+0xf0/0xf0 [12112.202117] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12112.204084] mdc_close+0x53b/0xd70 [mdc] [12112.206533] lmv_close+0x18d/0x490 [lmv] [12112.208008] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12112.210596] ll_md_real_close+0xa5/0x2a0 [lustre] [12112.212353] ll_md_close+0x274/0x800 [lustre] [12112.214295] ll_file_release+0x128/0xc50 [lustre] [12112.217986] __fput+0x10d/0x440 [12112.219001] ____fput+0x12/0x20 [12112.220798] task_work_run+0xc6/0x110 [12112.223247] exit_to_usermode_loop+0x1dd/0x1f0 [12112.225737] do_syscall_64+0x3d6/0x440 [12112.227279] entry_SYSCALL_64_after_hwframe+0x49/0xae [12112.229829] RIP: 0033:0x7f1a63983cd8 [12112.231258] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12112.238312] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12112.241812] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [12112.244391] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [12112.247278] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [12112.249193] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [12112.251550] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12120.066503] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [12120.073845] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12120.144850] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12120.157871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12120.179285] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12120.185752] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12120.199126] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12120.206101] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [12120.210022] RDX: 0000000000000d46 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [12120.218104] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [12120.229719] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [12120.236765] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12120.244025] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [12120.247764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12120.250773] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [12120.262151] Call Trace: [12120.265478] [12120.266134] ? show_regs.cold.9+0x22/0x2f [12120.271972] ? watchdog_timer_fn.cold.6+0x62/0x10a [12120.276048] ? __hrtimer_run_queues+0x1c2/0x4d0 [12120.278683] ? watchdog+0x40/0x40 [12120.282155] ? hrtimer_interrupt+0x128/0x2c0 [12120.283950] ? smp_apic_timer_interrupt+0xa6/0x270 [12120.286476] ? apic_timer_interrupt+0xf/0x20 [12120.288156] [12120.291171] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12120.295699] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12120.299935] do_raw_spin_lock+0x177/0x190 [12120.305869] _raw_spin_lock+0x21/0x30 [12120.309050] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12120.310921] obd_mod_free+0x46/0x140 [ptlrpc] [12120.314031] mdc_commit_open+0x91/0xb0 [mdc] [12120.315862] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12120.318455] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12120.323298] after_reply+0xe2c/0x1780 [ptlrpc] [12120.325257] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12120.328757] ? __next_timer_interrupt+0x160/0x160 [12120.332706] ptlrpcd+0xa82/0xe40 [ptlrpc] [12120.334961] ? do_wait_intr+0xf0/0xf0 [12120.339815] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12120.345013] kthread+0x1d1/0x200 [12120.346230] ? set_kthread_struct+0x70/0x70 [12120.350175] ret_from_fork+0x1f/0x30 [12128.058593] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:37206] [12128.068942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12128.110987] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12128.115554] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12128.126659] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12128.133711] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12128.179209] RSP: 0018:ffffc38541c37500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12128.186912] RAX: 0000000000006ad3 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12128.194046] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [12128.199602] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [12128.205648] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [12128.215095] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [12128.223397] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [12128.232769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12128.242858] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [12128.252216] Call Trace: [12128.254096] [12128.261712] ? show_regs.cold.9+0x22/0x2f [12128.264251] ? watchdog_timer_fn.cold.6+0x62/0x10a [12128.270860] ? __hrtimer_run_queues+0x1c2/0x4d0 [12128.275774] ? watchdog+0x40/0x40 [12128.278054] ? hrtimer_interrupt+0x128/0x2c0 [12128.287014] ? smp_apic_timer_interrupt+0xa6/0x270 [12128.295590] ? apic_timer_interrupt+0xf/0x20 [12128.298781] [12128.304076] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12128.312847] do_raw_spin_lock+0x177/0x190 [12128.317031] _raw_spin_lock+0x21/0x30 [12128.323609] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12128.329666] ? do_raw_spin_unlock+0x75/0x190 [12128.332056] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12128.336444] ? do_wait_intr+0xf0/0xf0 [12128.338619] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12128.341134] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12128.345306] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12128.348788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12128.352435] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12128.355891] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12128.358438] mdc_intent_lock+0x353/0x7e0 [mdc] [12128.362091] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12128.364708] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12128.367833] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12128.370500] lmv_intent_open+0x338/0x1150 [lmv] [12128.372630] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12128.379820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12128.383580] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12128.386585] ll_intent_lock+0xc5/0xaa0 [lustre] [12128.395532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12128.404100] ll_lookup_it+0xe0e/0x2270 [lustre] [12128.408688] ? do_raw_spin_unlock+0x75/0x190 [12128.424149] ? _raw_spin_unlock+0x12/0x30 [12128.427249] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12128.440008] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12128.452099] ll_atomic_open+0x36a/0xd70 [lustre] [12128.461424] lookup_open+0x356/0xb40 [12128.469077] do_last+0x313/0xfc0 [12128.485264] ? nd_jump_root+0xe5/0x160 [12128.497350] ? path_init+0x437/0x520 [12128.500959] path_openat+0xf7/0x500 [12128.502883] ? do_raw_spin_unlock+0x75/0x190 [12128.509993] do_filp_open+0x99/0x140 [12128.512357] ? getname_flags+0x6e/0x330 [12128.518441] ? __check_object_size+0xff/0x256 [12128.528466] ? do_raw_spin_unlock+0x75/0x190 [12128.534323] ? _raw_spin_unlock+0x12/0x30 [12128.540988] do_sys_openat2+0x2b4/0x410 [12128.545137] do_sys_open+0x73/0xa0 [12128.554841] __x64_sys_openat+0x24/0x30 [12128.558784] do_syscall_64+0xc1/0x440 [12128.566968] entry_SYSCALL_64_after_hwframe+0x49/0xae [12128.573151] RIP: 0033:0x7f1a63983332 [12128.577881] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12128.593176] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12128.602607] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [12128.610606] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [12128.622736] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [12128.630863] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [12128.637443] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [12132.327769] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10109s! [12132.349577] Showing busy workqueues and worker pools: [12132.356486] workqueue events: flags=0x0 [12132.359888] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12132.374039] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [12132.393174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12132.409321] pending: sec_gc_main [ptlrpc] [12132.421235] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12132.436313] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [12132.455953] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12132.467772] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [12132.499759] workqueue events_long: flags=0x0 [12132.505702] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12132.516423] pending: lu_objects_flush [obdclass] [12132.522622] workqueue events_unbound: flags=0x2 [12132.528607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12132.536730] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12132.554543] workqueue events_power_efficient: flags=0x80 [12132.559077] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12132.575024] pending: neigh_periodic_work, do_cache_clean [12132.586835] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12132.601770] pending: neigh_periodic_work, check_lifetime [12132.609885] workqueue mm_percpu_wq: flags=0x8 [12132.617095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12132.628205] pending: vmstat_update [12132.633938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12132.643798] pending: vmstat_update [12132.646506] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12132.650065] pending: vmstat_update [12132.651823] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12132.654798] pending: vmstat_update [12132.656571] workqueue writeback: flags=0x4a [12132.658317] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12132.665094] pending: wb_workfn, wb_workfn [12132.670329] workqueue xprtiod: flags=0xa [12132.676172] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12132.683118] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12132.695745] workqueue ipv6_addrconf: flags=0x40008 [12132.698753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12132.708572] pending: addrconf_verify_work [12132.715919] workqueue knbd0-recv: flags=0x1a [12132.720962] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12132.730922] in-flight: 647:recv_work [12132.735813] workqueue ptlrpc_pinger: flags=0x2 [12132.741075] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12132.753185] pending: ptlrpc_pinger_main [ptlrpc] [12132.767883] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12122s workers=2 idle: 652 [12136.057026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [writemany:37208] [12136.061261] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12136.089514] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12136.096660] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12136.102543] RIP: 0010:native_safe_halt+0x2e/0x50 [12136.105636] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12136.117645] RSP: 0018:ffffc3854145fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12136.122770] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12136.127673] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f8482134154 [12136.132440] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [12136.136876] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [12136.141696] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [12136.146004] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [12136.149144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12136.151553] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [12136.154461] Call Trace: [12136.155518] [12136.156424] ? show_regs.cold.9+0x22/0x2f [12136.158049] ? watchdog_timer_fn.cold.6+0x62/0x10a [12136.160095] ? __hrtimer_run_queues+0x1c2/0x4d0 [12136.162079] ? watchdog+0x40/0x40 [12136.163608] ? hrtimer_interrupt+0x128/0x2c0 [12136.165499] ? smp_apic_timer_interrupt+0xa6/0x270 [12136.167512] ? apic_timer_interrupt+0xf/0x20 [12136.169401] [12136.170148] ? native_safe_halt+0x2e/0x50 [12136.171849] kvm_wait+0xcd/0xe0 [12136.173134] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12136.175460] do_raw_spin_lock+0x177/0x190 [12136.177090] _raw_spin_lock+0x21/0x30 [12136.178749] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12136.181110] ? do_raw_spin_unlock+0x75/0x190 [12136.182953] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12136.185441] ? do_wait_intr+0xf0/0xf0 [12136.186948] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12136.189456] mdc_close+0x53b/0xd70 [mdc] [12136.191105] lmv_close+0x18d/0x490 [lmv] [12136.192860] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12136.195474] ll_md_real_close+0xa5/0x2a0 [lustre] [12136.197568] ll_md_close+0x274/0x800 [lustre] [12136.199534] ll_file_release+0x128/0xc50 [lustre] [12136.201662] __fput+0x10d/0x440 [12136.202968] ____fput+0x12/0x20 [12136.204473] task_work_run+0xc6/0x110 [12136.205999] exit_to_usermode_loop+0x1dd/0x1f0 [12136.208966] do_syscall_64+0x3d6/0x440 [12136.210769] entry_SYSCALL_64_after_hwframe+0x49/0xae [12136.213131] RIP: 0033:0x7f1a63983cd8 [12136.214904] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12136.223559] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12136.227057] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [12136.230411] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [12136.233655] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [12136.236909] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [12136.240150] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12140.056066] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37209] [12140.059358] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12140.079515] CPU: 1 PID: 37209 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12140.084796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12140.089112] RIP: 0010:native_safe_halt+0x2e/0x50 [12140.090478] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12140.098659] RSP: 0018:ffffc38542073ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12140.101604] RAX: 0000000000000001 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12140.104355] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f84820b4154 [12140.107064] RBP: ffff9f84820b4140 R08: 0000000000000000 R09: 0000000000000000 [12140.110502] R10: 0000000003caf565 R11: ffff9f84585e6000 R12: 0000000000080000 [12140.113518] R13: ffff9f84821b4140 R14: 000000000001e480 R15: ffff9f848209e480 [12140.116639] FS: 00007f1a64054740(0000) GS:ffff9f8482080000(0000) knlGS:0000000000000000 [12140.120428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12140.121939] CR2: 00005585f432b018 CR3: 00000001058b7006 CR4: 0000000000170ee0 [12140.124766] Call Trace: [12140.125702] [12140.126541] ? show_regs.cold.9+0x22/0x2f [12140.128345] ? watchdog_timer_fn.cold.6+0x62/0x10a [12140.130321] ? __hrtimer_run_queues+0x1c2/0x4d0 [12140.135264] ? watchdog+0x40/0x40 [12140.138645] ? hrtimer_interrupt+0x128/0x2c0 [12140.140590] ? smp_apic_timer_interrupt+0xa6/0x270 [12140.142668] ? apic_timer_interrupt+0xf/0x20 [12140.144793] [12140.145665] ? native_safe_halt+0x2e/0x50 [12140.147567] kvm_wait+0xcd/0xe0 [12140.149287] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12140.151371] do_raw_spin_lock+0x177/0x190 [12140.152910] _raw_spin_lock+0x21/0x30 [12140.155681] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12140.159893] ? do_raw_spin_unlock+0x75/0x190 [12140.163799] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12140.166590] ? do_wait_intr+0xf0/0xf0 [12140.167887] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12140.169855] mdc_close+0x53b/0xd70 [mdc] [12140.171570] lmv_close+0x18d/0x490 [lmv] [12140.175501] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12140.179807] ll_md_real_close+0xa5/0x2a0 [lustre] [12140.181460] ll_md_close+0x274/0x800 [lustre] [12140.184238] ll_file_release+0x128/0xc50 [lustre] [12140.188330] __fput+0x10d/0x440 [12140.191570] ____fput+0x12/0x20 [12140.197294] task_work_run+0xc6/0x110 [12140.199593] exit_to_usermode_loop+0x1dd/0x1f0 [12140.201305] do_syscall_64+0x3d6/0x440 [12140.202623] entry_SYSCALL_64_after_hwframe+0x49/0xae [12140.205358] RIP: 0033:0x7f1a63983cd8 [12140.207566] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12140.219308] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12140.224390] RAX: 0000000000000000 RBX: 0000000000009fe8 RCX: 00007f1a63983cd8 [12140.228976] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [12140.231629] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26228 R09: 00007f1a63c26260 [12140.234687] R10: 00007ffdbcfd1772 R11: 0000000000000246 R12: 0000000000000003 [12140.237648] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12148.058006] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpcd_rcv:2385] [12148.063589] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12148.085808] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12148.090494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12148.093608] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12148.095775] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12148.101880] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12148.104484] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [12148.107077] RDX: 0000000000003968 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [12148.109741] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [12148.112349] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [12148.114642] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12148.116852] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [12148.119324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12148.121058] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [12148.123757] Call Trace: [12148.124425] [12148.125077] ? show_regs.cold.9+0x22/0x2f [12148.126361] ? watchdog_timer_fn.cold.6+0x62/0x10a [12148.127898] ? __hrtimer_run_queues+0x1c2/0x4d0 [12148.129447] ? watchdog+0x40/0x40 [12148.130506] ? hrtimer_interrupt+0x128/0x2c0 [12148.131812] ? smp_apic_timer_interrupt+0xa6/0x270 [12148.133792] ? apic_timer_interrupt+0xf/0x20 [12148.134991] [12148.135644] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12148.138816] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12148.141268] do_raw_spin_lock+0x177/0x190 [12148.142988] _raw_spin_lock+0x21/0x30 [12148.144675] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12148.147187] obd_mod_free+0x46/0x140 [ptlrpc] [12148.149167] mdc_commit_open+0x91/0xb0 [mdc] [12148.150949] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12148.153538] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12148.157873] after_reply+0xe2c/0x1780 [ptlrpc] [12148.160536] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12148.163620] ? __next_timer_interrupt+0x160/0x160 [12148.165897] ptlrpcd+0xa82/0xe40 [ptlrpc] [12148.168422] ? do_wait_intr+0xf0/0xf0 [12148.169932] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12148.173809] kthread+0x1d1/0x200 [12148.175616] ? set_kthread_struct+0x70/0x70 [12148.178303] ret_from_fork+0x1f/0x30 [12160.057538] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:37206] [12160.064436] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12160.133196] CPU: 0 PID: 37206 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12160.150981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12160.167360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12160.169692] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12160.201953] RSP: 0018:ffffc38541c37500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12160.210504] RAX: 000000000000588d RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12160.223613] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482034154 [12160.232889] RBP: ffff9f8482034140 R08: 0000000000000000 R09: 0000000000000000 [12160.248633] R10: 0000000003caf6b1 R11: ffff9f8450859000 R12: 0000000000040000 [12160.269422] R13: ffff9f8482134140 R14: 000000000001e480 R15: ffff9f848201e480 [12160.280385] FS: 00007f1a64054740(0000) GS:ffff9f8482000000(0000) knlGS:0000000000000000 [12160.296180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12160.309715] CR2: 00007f2a75d20000 CR3: 000000011088e001 CR4: 0000000000170ef0 [12160.323210] Call Trace: [12160.324180] [12160.324923] ? show_regs.cold.9+0x22/0x2f [12160.335657] ? watchdog_timer_fn.cold.6+0x62/0x10a [12160.343422] ? __hrtimer_run_queues+0x1c2/0x4d0 [12160.353227] ? watchdog+0x40/0x40 [12160.354482] ? hrtimer_interrupt+0x128/0x2c0 [12160.359928] ? smp_apic_timer_interrupt+0xa6/0x270 [12160.361684] ? apic_timer_interrupt+0xf/0x20 [12160.371950] [12160.380642] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12160.388500] do_raw_spin_lock+0x177/0x190 [12160.389872] _raw_spin_lock+0x21/0x30 [12160.395491] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12160.412387] ? do_raw_spin_unlock+0x75/0x190 [12160.413887] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12160.422942] ? do_wait_intr+0xf0/0xf0 [12160.424302] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12160.429440] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12160.446466] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12160.448726] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12160.454775] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12160.462760] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12160.470482] mdc_intent_lock+0x353/0x7e0 [mdc] [12160.477053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12160.483641] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12160.491104] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12160.493067] lmv_intent_open+0x338/0x1150 [lmv] [12160.505499] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12160.507531] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12160.518241] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12160.519793] ll_intent_lock+0xc5/0xaa0 [lustre] [12160.529092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12160.533629] ll_lookup_it+0xe0e/0x2270 [lustre] [12160.539651] ? do_raw_spin_unlock+0x75/0x190 [12160.548871] ? _raw_spin_unlock+0x12/0x30 [12160.564109] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12160.569180] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12160.573537] ll_atomic_open+0x36a/0xd70 [lustre] [12160.587063] lookup_open+0x356/0xb40 [12160.591569] do_last+0x313/0xfc0 [12160.598053] ? nd_jump_root+0xe5/0x160 [12160.600473] ? path_init+0x437/0x520 [12160.612228] path_openat+0xf7/0x500 [12160.618540] ? do_raw_spin_unlock+0x75/0x190 [12160.626340] do_filp_open+0x99/0x140 [12160.627326] ? getname_flags+0x6e/0x330 [12160.628430] ? __check_object_size+0xff/0x256 [12160.631611] ? do_raw_spin_unlock+0x75/0x190 [12160.635649] ? _raw_spin_unlock+0x12/0x30 [12160.640379] do_sys_openat2+0x2b4/0x410 [12160.644666] do_sys_open+0x73/0xa0 [12160.649555] __x64_sys_openat+0x24/0x30 [12160.659406] do_syscall_64+0xc1/0x440 [12160.662919] entry_SYSCALL_64_after_hwframe+0x49/0xae [12160.668473] RIP: 0033:0x7f1a63983332 [12160.673653] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12160.688468] RSP: 002b:00007ffdbcfd1960 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12160.698834] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a63983332 [12160.702853] RDX: 0000000000000042 RSI: 00007ffdbcfd1a30 RDI: 00000000ffffff9c [12160.709879] RBP: 00007ffdbcfd1e30 R08: 0000000000000000 R09: 0000000000000000 [12160.721667] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffdbcfd1a30 [12160.734180] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [12162.675902] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [12162.679086] rcu: 3-...!: (6386401 ticks this GP) idle=826/1/0x4000000000000000 softirq=131754/131754 fqs=84 [12162.693582] (detected by 1, t=10140269 jiffies, g=269633, q=3497) [12162.700298] Sending NMI from CPU 1 to CPUs 3: [12162.713757] NMI backtrace for cpu 3 [12162.713766] CPU: 3 PID: 2385 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12162.713768] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12162.713770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12162.713772] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a c7 70 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12162.713774] RSP: 0018:ffffc38541af7bc8 EFLAGS: 00000206 [12162.713777] RAX: 0000000000000003 RBX: ffff9f8452a252e0 RCX: ffff9f848219e480 [12162.713779] RDX: 0000000000006cf6 RSI: 0000000000000003 RDI: ffff9f8452a252e0 [12162.713780] RBP: ffff9f84821b4140 R08: 0000000000000008 R09: 00000000000000ec [12162.713782] R10: ffffffff90e08b80 R11: ffffc38541af78f0 R12: 0000000000100000 [12162.713783] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12162.713785] FS: 0000000000000000(0000) GS:ffff9f8482180000(0000) knlGS:0000000000000000 [12162.713786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12162.713787] CR2: 00007f78bd239670 CR3: 0000000103016001 CR4: 0000000000170ee0 [12162.713789] Call Trace: [12162.713789] [12162.713790] ? show_regs.cold.9+0x22/0x2f [12162.713791] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12162.713792] ? nmi_cpu_backtrace_handler+0x15/0x40 [12162.713793] ? nmi_handle+0xbd/0x210 [12162.713794] ? default_do_nmi+0x4b/0x1b0 [12162.713795] ? do_nmi+0x280/0x380 [12162.713796] ? end_repeat_nmi+0x16/0x1f [12162.713797] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12162.713798] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12162.713799] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12162.713800] [12162.713801] do_raw_spin_lock+0x177/0x190 [12162.713802] _raw_spin_lock+0x21/0x30 [12162.713803] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12162.713804] obd_mod_free+0x46/0x140 [ptlrpc] [12162.713805] mdc_commit_open+0x91/0xb0 [mdc] [12162.713805] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12162.713806] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12162.713807] after_reply+0xe2c/0x1780 [ptlrpc] [12162.713808] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12162.713809] ? __next_timer_interrupt+0x160/0x160 [12162.713811] ptlrpcd+0xa82/0xe40 [ptlrpc] [12162.713811] ? do_wait_intr+0xf0/0xf0 [12162.713812] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12162.713813] kthread+0x1d1/0x200 [12162.713814] ? set_kthread_struct+0x70/0x70 [12162.713815] ret_from_fork+0x1f/0x30 [12162.714457] rcu: rcu_sched kthread starved for 10139915 jiffies! g269633 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [12162.907450] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12162.913411] rcu: RCU grace-period kthread stack dump: [12162.918479] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12162.925010] Call Trace: [12162.926689] __schedule+0x351/0xcb0 [12162.927819] schedule+0xc0/0x180 [12162.929525] schedule_timeout+0xb4/0x190 [12162.932471] ? do_raw_spin_unlock+0x75/0x190 [12162.934687] ? __next_timer_interrupt+0x160/0x160 [12162.936359] ? prepare_to_swait_event+0xf3/0x140 [12162.938282] rcu_gp_kthread+0x8dd/0x15b0 [12162.940876] ? force_qs_rnp+0x2c0/0x2c0 [12162.942792] kthread+0x1d1/0x200 [12162.945485] ? set_kthread_struct+0x70/0x70 [12162.949565] ret_from_fork+0x1f/0x30 [12162.951653] rcu: Stack dump where RCU GP kthread last ran: [12162.953260] Sending NMI from CPU 1 to CPUs 2: [12162.958857] NMI backtrace for cpu 2 [12162.958864] CPU: 2 PID: 37208 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12162.958866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12162.958868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12162.958870] Code: 4c 03 2c c5 80 58 c8 90 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12162.958873] RSP: 0018:ffffc3854145faf0 EFLAGS: 00000202 [12162.958875] RAX: 0000000000001b09 RBX: ffff9f8452a252e0 RCX: 0000000000000001 [12162.958877] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9f8482134154 [12162.958879] RBP: ffff9f8482134140 R08: 0000000000000000 R09: 0000000000000000 [12162.958880] R10: 0000000003caf60b R11: ffff9f845851f000 R12: 00000000000c0000 [12162.958882] R13: ffff9f84820b4140 R14: 000000000001e480 R15: ffff9f848211e480 [12162.958883] FS: 00007f1a64054740(0000) GS:ffff9f8482100000(0000) knlGS:0000000000000000 [12162.958885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12162.958886] CR2: 00005585f4346528 CR3: 0000000111a61003 CR4: 0000000000170ee0 [12162.958887] Call Trace: [12162.958888] [12162.958889] ? show_regs.cold.9+0x22/0x2f [12162.958890] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12162.958892] ? nmi_cpu_backtrace_handler+0x15/0x40 [12162.958893] ? nmi_handle+0xbd/0x210 [12162.958894] ? default_do_nmi+0x4b/0x1b0 [12162.958895] ? do_nmi+0x280/0x380 [12162.958896] ? end_repeat_nmi+0x16/0x1f [12162.958898] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12162.958899] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12162.958901] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12162.958902] [12162.958903] do_raw_spin_lock+0x177/0x190 [12162.958904] _raw_spin_lock+0x21/0x30 [12162.958905] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12162.958906] ? do_raw_spin_unlock+0x75/0x190 [12162.958907] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12162.958908] ? do_wait_intr+0xf0/0xf0 [12162.958909] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12162.958910] mdc_close+0x53b/0xd70 [mdc] [12162.958912] lmv_close+0x18d/0x490 [lmv] [12162.958913] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12162.958914] ll_md_real_close+0xa5/0x2a0 [lustre] [12162.958915] ll_md_close+0x274/0x800 [lustre] [12162.958916] ll_file_release+0x128/0xc50 [lustre] [12162.958917] __fput+0x10d/0x440 [12162.958918] ____fput+0x12/0x20 [12162.958919] task_work_run+0xc6/0x110 [12162.958920] exit_to_usermode_loop+0x1dd/0x1f0 [12162.958921] do_syscall_64+0x3d6/0x440 [12162.958922] entry_SYSCALL_64_after_hwframe+0x49/0xae [12162.958924] RIP: 0033:0x7f1a63983cd8 [12162.958925] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12162.958928] RSP: 002b:00007ffdbcfd19c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12162.958930] RAX: 0000000000000000 RBX: 0000000000001030 RCX: 00007f1a63983cd8 [12162.958932] RDX: 0000000000000038 RSI: 00007ffdbcfd1e30 RDI: 0000000000000003 [12162.958933] RBP: 00007ffdbcfd1e30 R08: 00007f1a63c26224 R09: 00007f1a63c26260 [12162.958935] R10: 00007ffdbcfd1773 R11: 0000000000000246 R12: 0000000000000003 [12162.958936] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12163.041526] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10140s! [12163.206517] Showing busy workqueues and worker pools: [12163.209256] workqueue events: flags=0x0 [12163.214554] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12163.221853] pending: ldlm_pools_recalc_task [ptlrpc], push_to_pool [12163.228227] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12163.236464] pending: sec_gc_main [ptlrpc] [12163.239687] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12163.246553] pending: cpu_latency_work [ptlrpc], nfs4_renew_state [nfsv4], key_garbage_collector [12163.253785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12163.266642] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], osc_grant_work_handler [osc] [12163.278206] workqueue events_long: flags=0x0 [12163.280374] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12163.283725] pending: lu_objects_flush [obdclass] [12163.292333] workqueue events_unbound: flags=0x2 [12163.293799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12163.297015] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12163.312026] workqueue events_power_efficient: flags=0x80 [12163.317411] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12163.326665] pending: neigh_periodic_work, do_cache_clean [12163.330385] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12163.338187] pending: neigh_periodic_work, check_lifetime [12163.341537] workqueue mm_percpu_wq: flags=0x8 [12163.343650] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12163.359154] pending: vmstat_update [12163.360764] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12163.371286] pending: vmstat_update [12163.376192] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12163.380241] pending: vmstat_update [12163.382161] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12163.385544] pending: vmstat_update [12163.388215] workqueue writeback: flags=0x4a [12163.394106] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12163.398449] pending: wb_workfn, wb_workfn [12163.404553] workqueue xprtiod: flags=0xa [12163.408429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12163.412749] pending: xs_stream_data_receive_workfn, xs_error_handle, xprt_autoclose [12163.423247] workqueue ipv6_addrconf: flags=0x40008 [12163.425749] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12163.431628] pending: addrconf_verify_work [12163.434508] workqueue knbd0-recv: flags=0x1a [12163.438797] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12163.443416] in-flight: 647:recv_work [12163.449237] workqueue ptlrpc_pinger: flags=0x2 [12163.450900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12163.458425] pending: ptlrpc_pinger_main [ptlrpc] [12163.467982] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12153s workers=2 idle: 652